Mirantis Container Cloud Documentation¶

This documentation is intended to help operators understand the core concepts of the product.

The information provided in this documentation set is being constantly improved and amended based on the feedback and kind requests from our software consumers. This documentation set outlines description of the features supported within three latest Container Cloud minor releases and their supported Cluster releases, with a corresponding note ^{Available since <release-version>}.

The following table lists the guides included in the documentation set you are reading:

Guides list¶
Guide	Purpose
Reference Architecture	Learn the fundamentals of Container Cloud reference architecture to plan your deployment.
Deployment Guide	Deploy Container Cloud of a preferred configuration using supported deployment profiles tailored to the demands of specific business cases.
Operations Guide	Deploy and operate the Container Cloud managed clusters.
Release Compatibility Matrix	Deployment compatibility of the Container Cloud components versions for each product release.
Release Notes	Learn about new features and bug fixes in the current Container Cloud version as well as in the Container Cloud minor releases.

Intended audience¶

This documentation assumes that the reader is familiar with network and cloud concepts and is intended for the following users:

Infrastructure Operator
- Is member of the IT operations team
- Has working knowledge of Linux, virtualization, Kubernetes API and CLI, and OpenStack to support the application development team
- Accesses Mirantis Container Cloud and Kubernetes through a local machine or web UI
- Provides verified artifacts through a central repository to the Tenant DevOps engineers
Tenant DevOps engineer
- Is member of the application development team and reports to line-of-business (LOB)
- Has working knowledge of Linux, virtualization, Kubernetes API and CLI to support application owners
- Accesses Container Cloud and Kubernetes through a local machine or web UI
- Consumes artifacts from a central repository approved by the Infrastructure Operator

Conventions¶

This documentation set uses the following conventions in the HTML format:

Documentation conventions¶
Convention	Description
boldface font	Inline CLI tools and commands, titles of the procedures and system response examples, table titles.
`monospaced` font	Files names and paths, Helm charts parameters and their values, names of packages, nodes names and labels, and so on.
italic font	Information that distinguishes some concept or term.
Links	External links and cross-references, footnotes.
Main menu > menu item	GUI elements that include any part of interactive user interface and menu navigation.
^Superscript	Some extra, brief information. For example, if a feature is available from a specific release or if a feature is in the Technology Preview development stage.
Note The Note block	Messages of a generic meaning that may be useful to the user.
Caution The Caution block	Information that prevents a user from mistakes and undesirable consequences when following the procedures.
Warning The Warning block	Messages that include details that can be easily missed, but should not be ignored by the user and are valuable before proceeding.
See also The See also block	List of references that may be helpful for understanding of some related tools, concepts, and so on.
Learn more The Learn more block	Used in the Release Notes to wrap a list of internal references to the reference architecture, deployment and operation procedures specific to a newly implemented product feature.

Technology Preview features¶

A Technology Preview feature provides early access to upcoming product innovations, allowing customers to experiment with the functionality and provide feedback.

Technology Preview features may be privately or publicly available but neither are intended for production use. While Mirantis will provide assistance with such features through official channels, normal Service Level Agreements do not apply.

As Mirantis considers making future iterations of Technology Preview features generally available, we will do our best to resolve any issues that customers experience when using these features.

During the development of a Technology Preview feature, additional components may become available to the public for evaluation. Mirantis cannot guarantee the stability of such features. As a result, if you are using Technology Preview features, you may not be able to seamlessly update to subsequent product releases, as well as upgrade or migrate to the functionality that has not been announced as full support yet.

Mirantis makes no guarantees that Technology Preview features will graduate to generally available features.

Documentation history¶

The documentation set refers to Mirantis Container Cloud GA as to the latest released GA version of the product. For details about the Container Cloud GA minor releases dates, refer to Container Cloud releases.

Product Overview¶

Mirantis Container Cloud enables you to ship code faster by enabling speed with choice, simplicity, and security. Through a single pane of glass you can deploy, manage, and observe Kubernetes clusters on bare metal infrastructure.

The list of the most common use cases includes:

Kubernetes cluster lifecycle management: The consistent lifecycle management of a single Kubernetes cluster is a complex task on its own that is made infinitely more difficult when you have to manage multiple clusters across different platforms spread across the globe. Mirantis Container Cloud provides a single, centralized point from which you can perform full lifecycle management of your container clusters, including automated updates and upgrades.
Highly regulated industries: Regulated industries need a fine level of access control granularity, high security standards and extensive reporting capabilities to ensure that they can meet and exceed the security standards and requirements. Mirantis Container Cloud provides for a fine-grained Role Based Access Control (RBAC) mechanism and easy integration and federation to existing identity management systems (IDM).
Logging, monitoring, alerting: A complete operational visibility is required to identify and address issues in the shortest amount of time – before the problem becomes serious. Mirantis StackLight is the proactive monitoring, logging, and alerting solution designed for large-scale container and cloud observability with extensive collectors, dashboards, trend reporting and alerts.
Storage: Cloud environments require a unified pool of storage that can be scaled up by simply adding storage server nodes. Ceph is a unified, distributed storage system designed for excellent performance, reliability, and scalability. Deploy Ceph utilizing Rook to provide and manage a robust persistent storage that can be used by Kubernetes workloads on the baremetal-based clusters.
Security: Security is a core concern for all enterprises, especially with more of our systems being exposed to the Internet as a norm. Mirantis Container Cloud provides for a multi-layered security approach that includes effective identity management and role based authentication, secure out of the box defaults and extensive security scanning and monitoring during the development process.
5G and Edge: The introduction of 5G technologies and the support of Edge workloads requires an effective multi-tenant solution to manage the underlying container infrastructure. Mirantis Container Cloud provides for a full stack, secure, multi-cloud cluster management and Day-2 operations solution.

Reference Architecture¶

This Reference Architecture is now part of MOSK Reference Architecture.

Overview¶

This Reference Architecture is now part of MOSK Reference Architecture.

Bare metal¶

The subsections of this section were moved to MOSK documentation : Bare metal, Networking, Storage, and Automatic upgrade of a host operating system.

Bare metal components¶

This section was moved to MOSK documentation: Bare metal components.

Bare metal networking¶

The subsections of this section were moved to MOSK documentation: Networking.

IP Address Management¶

This section was moved to MOSK documentation: IP Address Management.

Management cluster networking¶

This section was moved to MOSK documentation: Management cluster networking.

Managed cluster networking¶

This section was moved to MOSK documentation: Cluster networking.

Host networking¶

This section was moved to MOSK documentation: Network types.

Storage¶

The subsections of this section were moved to MOSK documentation: Storage.

Overview¶

This section was moved to MOSK documentation: Ceph overview.

Limitations¶

This section was moved to MOSK documentation: Ceph limitations.

Addressing storage devices¶

This section was moved to MOSK documentation: Addressing storage devices.

Extended hardware configuration¶

This section was moved to MOSK documentation: Extended hardware configuration.

Automatic upgrade of a host operating system¶

This section was moved to MOSK documentation: Automatic upgrade of a host operating system.

Built-in load balancing¶

This section was moved to MOSK documentation: Built-in load balancing.

Kubernetes lifecycle management¶

The subsections of this section were moved to MOSK documentation: Kubernetes lifecycle management.

LCM custom resources¶

This section was moved to MOSK documentation: LCM custom resources.

LCM Controller¶

This section was moved to MOSK documentation: LCM Controller.

LCM Agent¶

This section was moved to MOSK documentation: LCM Agent.

Helm Controller¶

This section was moved to MOSK documentation: Helm Controller.

Identity and access management¶

The subsections of this section were moved to MOSK documentation: Identity and access management.

External identity provider integration¶

This section was moved to MOSK documentation: External identity provider integration.

Authentication and authorization¶

The subsections of this section were moved to MOSK documentation: IAM - Authentication and authorization.

Implementation flow¶

This section was moved to MOSK documentation: IAM - Implementation flow.

Kubernetes CLI authentication flow¶

This section was moved to MOSK documentation: Kubernetes CLI authentication flow.

Monitoring¶

The subsections of this section were moved to MOSK documentation: Logging, monitoring, and alerting.

Deployment architecture¶

This section was moved to MOSK documentation: StackLight deployment architecture.

Authentication flow¶

This section was moved to MOSK documentation: StackLight authentication flow.

Supported features¶

This section was moved to MOSK documentation: StackLight supported features.

Monitored components¶

This section was moved to MOSK documentation: StackLight monitored components.

Storage-based log retention strategy¶

This section was moved to MOSK documentation: Storage-based log retention strategy.

Outbound cluster metrics¶

This section was moved to MOSK documentation: Outbound cluster metrics.

StackLight proxy¶

This section was moved to MOSK documentation: StackLight proxy.

Requirements¶

The subsections of this section were moved to MOSK documentation: Requirements.

Reference hardware configuration¶

This section was moved to MOSK documentation: MOSK cluster hardware requirements.

System requirements for the seed node¶

This section was moved to MOSK documentation: System requirements for the seed node.

Network fabric¶

This section was moved to MOSK documentation: Physical networks layout.

DHCP range requirements for PXE¶

This section was moved to MOSK documentation: DHCP range requirements for PXE.

Management cluster storage¶

This section was moved to MOSK documentation: Management cluster storage.

Proxy and cache support¶

This section was moved to MOSK documentation: Proxy and cache support.

MKE API limitations¶

This section was moved to MOSK documentation: MKE API limitations.

MKE configuration management¶

This section was moved to MOSK documentation: MKE configuration management.

Deployment Guide¶

This guide was moved to MOSK Deployment Guide: Deploy a management cluster.

Deploy a Container Cloud management cluster¶

The subsections of this section were moved to MOSK Deployment Guide: Deploy a management cluster.

Introduction¶

This section was moved to MOSK Deployment Guide: Deploy a management cluster - Introduction.

Overview of the deployment workflow¶

This section was moved to MOSK Deployment Guide: Deploy a management cluster - Overview of the deployment workflow.

Set up a bootstrap cluster¶

This section was moved to MOSK Deployment Guide: Set up a bootstrap cluster.

Deploy a management cluster using the Container Cloud API¶

This section was moved to MOSK Deployment Guide: Deploy a management cluster.

Configure a bare metal deployment¶

The subsections of this section were moved to MOSK Deployment Guide: Deploy a management cluster - Configure a bare metal deployment.

Configure BIOS on a bare metal host¶

This section was moved to MOSK Deployment Guide: Configure bare metal settings - Configure BIOS on a bare metal host.

Customize the default bare metal host profile¶

This section was moved to MOSK Deployment Guide: Configure bare metal settings - Customize the default bare metal host profile.

Configure NIC bonding¶

This section was moved to MOSK Deployment Guide: Configure bare metal settings - Configure NIC bonding.

Separate PXE and management networks¶

This section was moved to MOSK Deployment Guide: Configure bare metal settings - Separate PXE and management networks.

Configure multiple DHCP address ranges¶

This section was moved to MOSK Deployment Guide: Configure bare metal settings - Configure multiple DHCP address ranges.

Enable dynamic IP allocation¶

This section was moved to MOSK Deployment Guide: Configure bare metal settings - Enable dynamic IP allocation.

Set a custom external IP address for the DHCP service¶

This section was moved to MOSK Deployment Guide: Configure bare metal settings - Set a custom external IP address for the DHCP service.

Configure optional cluster settings¶

This section was moved to MOSK Deployment Guide: Deploy a management cluster - Configure optional cluster settings.

Post-deployment steps¶

This section was moved to MOSK Deployment Guide: Deploy a management cluster - Post-deployment steps.

Troubleshooting¶

The subsections of this section were moved to MOSK Troubleshooting Guide: Troubleshoot a management cluster bootstrap.

Troubleshoot the bootstrap region creation¶

This section was moved to MOSK Troubleshooting Guide: Troubleshoot a management cluster bootstrap - Troubleshoot the bootstrap region creation.

Troubleshoot machines creation¶

This section was moved to MOSK Troubleshooting Guide: Troubleshoot a management cluster bootstrap - Troubleshoot machines creation.

Troubleshoot deployment stages¶

This section was moved to MOSK Troubleshooting Guide: Troubleshoot a management cluster bootstrap - Troubleshoot deployment stages.

Collect the bootstrap logs¶

This section was moved to MOSK Troubleshooting Guide: Troubleshoot a management cluster bootstrap - Collect the bootstrap logs.

Requirements for a MITM proxy¶

This section was moved to MOSK Deployment Guide: Requirements for a MITM proxy.

Create initial users after a management cluster bootstrap¶

This section was moved to MOSK Deployment Guide: Create initial users after a management cluster bootstrap.

Troubleshooting¶

The subsections of this section were moved to MOSK Troubleshooting Guide: Troubleshoot the bootstrap node configuration.

Troubleshoot the bootstrap node configuration¶

The subsections of this section were moved to MOSK Troubleshooting Guide: Troubleshoot the bootstrap node configuration.

DNS settings¶

This section was moved to MOSK Troubleshooting Guide: Troubleshoot the bootstrap node configuration - DNS settings.

Default network addresses¶

This section was moved to MOSK Troubleshooting Guide: Troubleshoot the bootstrap node configuration - Default network addresses.

Configure external identity provider for IAM¶

The subsections of this section were moved to MOSK Deployment Guide: Configure external identity provider for IAM.

Configure LDAP for IAM¶

This section was moved to MOSK Deployment Guide: Configure external identity provider for IAM - Configure LDAP for IAM.

Configure Google OAuth IdP for IAM¶

This section was moved to MOSK Deployment Guide: Configure external identity provider for IAM - Configure Google OAuth IdP for IAM.

Operations Guide¶

This guide is now part of MOSK Operations Guide.

Mirantis Container Cloud CLI¶

This section was moved to MOSK documentation: Container Cloud CLI.

Create and operate managed clusters¶

Note

This tutorial applies only to the Container Cloud web UI users with the m:kaas:namespace@operator or m:kaas:namespace@writer access role assigned by the Infrastructure Operator. To add a bare metal host, the m:kaas@operator or m:kaas:namespace@bm-pool-operator role is required.

After you deploy the Mirantis Container Cloud management cluster, you can start creating managed clusters depending on your cloud needs.

The deployment procedure is performed using the Container Cloud web UI and comprises the following steps:

Create a dedicated non-default project for managed clusters.
Create and configure bare metal hosts with corresponding labels for machines such as worker, manager, or storage.
Create an initial cluster configuration.
Add the required amount of machines with the corresponding configuration to the managed cluster.
Add a Ceph cluster.

Note

The Container Cloud web UI communicates with Keycloak to authenticate users. Keycloak is exposed using HTTPS with self-signed TLS certificates that are not trusted by web browsers.

To use your own TLS certificates for Keycloak, refer to Configure TLS certificates for cluster applications.

Create a project for managed clusters¶

This section was moved to MOSK Deployment Guide: Create a project for managed clusters.

Generate a kubeconfig for a managed cluster using API¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Getting access - Generate a kubeconfig for a cluster using API.

Create and operate a baremetal-based managed cluster¶

The subsections of this section were moved to Mirantis OpenStack for Kubernetes documentation: Create a managed cluster.

Add a bare metal host¶

The subsections of this section were moved to MOSK Deployment Guide: Add a bare metal host.

Add a bare metal host using web UI¶

This section was moved to MOSK Deployment Guide: Add a bare metal host using web UI.

Add a bare metal host using CLI¶

This section was moved to MOSK Deployment Guide: Add a bare metal host using web CLI.

Create a custom bare metal host profile¶

The subsections of this section were moved to MOSK Deployment Guide: Create MOSK host profiles.

Default configuration of the host system storage¶

This section was moved to MOSK Deployment Guide: Default configuration of the host system storage.

Wipe a device or partition¶

Available since 2.26.0 (17.1.0 and 16.1.0)

This section was moved to MOSK Deployment Guide: Wipe a device or partition.

Create a custom host profile¶

This section was moved to MOSK Deployment Guide: Create a custom host profile.

Configure Ceph disks in a host profile¶

This section was moved to MOSK Deployment Guide: Configure Ceph disks in a host profile.

Enable huge pages¶

This section was moved to MOSK Deployment Guide: Enable huge pages.

Configure RAID support¶

Caution

This feature is available as Technology Preview. Use such configuration for testing and evaluation purposes only. For the Technology Preview feature definition, refer to Technology Preview features.

The subsections of this section were moved to MOSK Deployment Guide: Configure RAID support.

Create an LVM software RAID level 1 (raid1)¶

Caution

This feature is available as Technology Preview. Use such configuration for testing and evaluation purposes only. For the Technology Preview feature definition, refer to Technology Preview features.

This section was moved to MOSK Deployment Guide: Create an LVM software RAID level 1 (raid1).

Create an mdadm software RAID level 1 (raid1)¶

Caution

This feature is available as Technology Preview. Use such configuration for testing and evaluation purposes only. For the Technology Preview feature definition, refer to Technology Preview features.

This section was moved to MOSK Deployment Guide: Create an mdadm software RAID level 1 (raid1).

Create an mdadm software RAID level 10 (raid10)¶

Technology Preview

This section was moved to MOSK Deployment Guide: Create an mdadm software RAID level 10 (raid10).

Add a managed baremetal cluster¶

The subsections of this section were moved to Mirantis OpenStack for Kubernetes documentation: Create a managed cluster.

Create a cluster using web UI¶

This section was moved to MOSK Deployment Guide: Create a cluster using web UI.

Workflow of network interface naming¶

This section was moved to MOSK Deployment Guide: Workflow of network interface naming.

Create subnets¶

The subsections of this section were moved to Mirantis OpenStack for Kubernetes documentation: Create subnets.

Service labels and their life cycle¶

This section was moved to MOSK Deployment Guide: Service labels and their life cycle.

MetalLB configuration guidelines for subnets¶

This section was moved to MOSK Deployment Guide: MetalLB configuration guidelines for subnets.

Configure MetalLB¶

This section was moved to MOSK Deployment Guide: Configure MetalLB.

Configure node selector for MetalLB speaker¶

This section was moved to MOSK Deployment Guide: Configure node selector for MetalLB speaker.

Create subnets for a managed cluster using web UI¶

This section was moved to MOSK Deployment Guide: Create subnets for a managed cluster using web UI.

Create subnets for a managed cluster using CLI¶

This section was moved to MOSK Deployment Guide: Create subnets for a managed cluster using CLI.

Automate multiple subnet creation using SubnetPool¶

Unsupported since 2.28.0 (17.3.0 and 16.3.0)

This section was moved to MOSK Deployment Guide: Automate multiple subnet creation using SubnetPool network interface naming.

Create L2 templates¶

The subsections of this section were moved to Mirantis OpenStack for Kubernetes documentation: Create L2 templates.

L2 template example with bonds and bridges¶

This section was moved to MOSK Deployment Guide: L2 template example with bonds and bridges.

L2 template example for automatic multiple subnet creation¶

Unsupported since 2.28.0 (17.3.0 and 16.3.0)

This section was moved to MOSK Deployment Guide: L2 template example for automatic multiple subnet creation.

Configure BGP announcement for cluster API LB address¶

This section was moved to MOSK Deployment Guide: Configure BGP announcement for cluster API LB address.

Add a machine¶

The subsections of this section were moved to MOSK Deployment Guide: Add a machine.

Create a machine using web UI¶

This section was moved to MOSK Deployment Guide: Add a machine using web UI.

Create a machine using CLI¶

The subsections of this section were moved to MOSK Deployment Guide: Add a machine.

Deploy a machine to a specific bare metal host¶

This section was moved to MOSK Deployment Guide: Deploy a machine to a specific bare metal host.

Assign L2 templates to machines¶

This section was moved to MOSK Deployment Guide: Assign L2 templates to machines.

Override network interfaces naming and order¶

This section was moved to MOSK Deployment Guide: Override network interfaces naming and order.

Manually allocate IP addresses for bare metal hosts¶

Available since Cluster releases 16.0.0 and 17.0.0 as TechPreview and since 16.1.0 and 17.1.0 as GA

This section was moved to MOSK Deployment Guide: Manually allocate IP addresses for bare metal hosts.

Add a Ceph cluster¶

The subsections of this section were moved to MOSK Deployment Guide: Add a Ceph cluster.

Add a Ceph cluster using web UI¶

This section was moved to MOSK Deployment Guide: Add a Ceph cluster.

Add a Ceph cluster using CLI¶

This section was moved to MOSK Deployment Guide: Add a Ceph cluster.

Example of a complete L2 templates configuration for cluster creation¶

This section was moved to MOSK documentation: Deploy a managed cluster - Example of a complete template configuration for cluster creation.

The subsections of this section were moved to Mirantis OpenStack for Kubernetes documentation: Operations Guide.

Manage an existing bare metal cluster¶

The subsections of this section were moved to Mirantis OpenStack for Kubernetes documentation: Bare metal operations.

Expand IP addresses capacity in an existing cluster¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Bare metal operations - Expand IP addresses capacity in an existing cluster.

Manage machines of a bare metal cluster¶

The subsections of this section were moved to Mirantis OpenStack for Kubernetes documentation: Bare metal operations.

Upgrade an operating system distribution¶

Available since 14.0.1 and 15.0.1 for MOSK 23.2

This section was moved to Mirantis OpenStack for Kubernetes documentation: Bare metal operations.

Remove old Ubuntu kernel packages¶

Available since 2.25.0

This section was moved to Mirantis OpenStack for Kubernetes documentation: Bare metal operations.

Modify network configuration on an existing machine¶

TechPreview

This section was moved to Mirantis OpenStack for Kubernetes documentation: Bare metal operations.

Change a user name and password for a bare metal host¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Bare metal operations.

Manage Ceph¶

The subsections of this section were moved to Mirantis OpenStack for Kubernetes documentation: Manage Ceph.

Ceph advanced configuration¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Ceph advanced configuration.

Ceph default configuration options¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Ceph advanced configuration.

Automated Ceph LCM¶

The subsections of this section were moved to MOSK documentation: Automated Ceph LCM.

High-level workflow of Ceph OSD or node removal¶

The subsections of this section were moved to Mirantis OpenStack for Kubernetes documentation: High-level workflow of Ceph OSD or node removal.

Creating a Ceph OSD removal request¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Creating a Ceph OSD removal request.

KaaSCephOperationRequest OSD removal specification¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: KaaSCephOperationRequest OSD removal specification.

KaaSCephOperationRequest OSD removal status¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: KaaSCephOperationRequest OSD removal status.

Add, remove, or reconfigure Ceph nodes¶

This section was moved to MOSK documentation: Ceph operations - Add, remove, or reconfigure Ceph nodes.

Add, remove, or reconfigure Ceph OSDs¶

This section was moved to MOSK documentation: Ceph operations - Add, remove, or reconfigure Ceph OSDs.

Add, remove, or reconfigure Ceph OSDs with metadata devices¶

This section was moved to MOSK documentation: Automated Ceph LCM - Add, remove, or reconfigure Ceph OSDs with metadata devices.

Replace a failed Ceph OSD¶

This section was moved to MOSK documentation: Automated Ceph LCM - Replace a failed Ceph OSD.

Replace a failed Ceph OSD with a metadata device¶

The subsections of this section were moved to MOSK documentation: Automated Ceph LCM - Replace a failed Ceph OSD with a metadata device.

Replace a failed Ceph OSD with a metadata device as a logical volume path¶

This section was moved to MOSK documentation: Automated Ceph LCM - Replace a failed Ceph OSD with a metadata device as a logical volume path.

Replace a failed Ceph OSD disk with a metadata device as a device name¶

This section was moved to MOSK documentation: Automated Ceph LCM - Replace a failed Ceph OSD disk with a metadata device as a device name.

Replace a failed metadata device¶

This section was moved to MOSK documentation: Automated Ceph LCM - Replace a failed metadata device.

Replace a failed Ceph node¶

This section was moved to MOSK documentation: Automated Ceph LCM - Replace a failed Ceph node.

Migrate Ceph cluster to address storage devices using by-id¶

This section was moved to MOSK documentation: Ceph operations - Migrate Ceph cluster to address storage devices using by-id.

Obtain a by-id symlink of a storage device¶

This section was moved to MOSK documentation: Ceph operations - Obtain a by-id symlink of a storage device.

Increase Ceph cluster storage size¶

This section was moved to MOSK documentation: Ceph operations - Increase Ceph cluster storage size.

Move a Ceph Monitor daemon to another node¶

This section was moved to MOSK documentation: Ceph operations - Move a Ceph Monitor daemon to another node.

Migrate a Ceph Monitor before machine replacement¶

This section was moved to MOSK documentation: Ceph operations - Migrate a Ceph Monitor before machine replacement.

Enable Ceph RGW Object Storage¶

This section was moved to MOSK documentation: Ceph operations - Enable Ceph RGW Object Storage.

Enable multisite for Ceph RGW Object Storage¶

This section was moved to MOSK documentation: Ceph operations - Enable multisite for Ceph RGW Object Storage.

Manage Ceph RBD or CephFS clients and RGW users¶

Available since 2.21.0 for non-MOSK clusters

The subsections of this section were moved to MOSK documentation: Ceph operations - Manage Ceph RBD or CephFS clients and RGW users.

Manage Ceph RBD or CephFS clients¶

This section was moved to MOSK documentation: Ceph operations - Manage Ceph RBD or CephFS clients.

Manage Ceph Object Storage users¶

This section was moved to MOSK documentation: Ceph operations - Manage Ceph Object Storage users.

Set an Amazon S3 bucket policy¶

The subsections of this section were moved to MOSK documentation: Ceph operations - Set an Amazon S3 bucket policy.

Create Ceph Object Storage users¶

This section was moved to MOSK documentation: Ceph operations Create Ceph Object Storage users.

Set a bucket policy for a Ceph Object Storage user¶

This section was moved to MOSK documentation: Ceph operations Set a bucket policy for a Ceph Object Storage user.

Verify Ceph¶

The subsections of this section were moved to Mirantis OpenStack for Kubernetes documentation: Ceph operations - Verify Ceph.

Verify the Ceph core services¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Ceph operations - Verify the Ceph core services.

Verify rook-discover¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Ceph operations - Verify rook-discover.

Verify Ceph cluster state through CLI¶

The subsections of this section were moved to Mirantis OpenStack for Kubernetes documentation: Ceph operations - Verify Ceph cluster state through CLI.

Verify Ceph cluster state¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Ceph operations - Verify Ceph cluster state.

KaaSCephCluster.status description¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Ceph operations - KaaSCephCluster.status description.

View Ceph cluster summary through the Container Cloud web UI¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Ceph operations - View Ceph cluster summary through the Container Cloud web UI.

Verify Ceph Controller and Rook¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Ceph operations - Verify Ceph Controller and Rook.

Enable Ceph tolerations and resources management¶

The subsections of this section were moved to Mirantis OpenStack for Kubernetes documentation: Ceph operations - Enable Ceph tolerations and resources management.

Enable Ceph tolerations and resources management¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Ceph operations - Enable Ceph tolerations and resources management.

Verify Ceph tolerations and resources management¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Ceph operations - Verify Ceph tolerations and resources management.

Enable Ceph multinetwork¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Ceph operations - Enable Ceph multinetwork.

Enable TLS for Ceph public endpoints¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Ceph operations - Configure Ceph Object Gateway TLS.

Enable Ceph RBD mirroring¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Ceph operations - Enable Ceph RBD mirroring.

Enable Ceph Shared File System (CephFS)¶

Available since 2.22.0 as GA

This section was moved to Mirantis OpenStack for Kubernetes documentation: Ceph operations - Enable Ceph Shared File System (CephFS).

Calculate target ratio for Ceph pools¶

This section was moved to MOSK documentation: Ceph operations - Calculate target ratio for Ceph pools.

Specify placement of Ceph cluster daemons¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Ceph operations - Specify placement of Ceph cluster daemons.

Migrate Ceph pools from one failure domain to another¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Ceph operations - Migrate Ceph pools from one failure domain to another.

Enable periodic Ceph performance testing¶

TechPreview

The subsections of this section were moved to Mirantis OpenStack for Kubernetes documentation: Enable periodic Ceph performance testing.

Create a Ceph performance test request¶

TechPreview

This section was moved to Mirantis OpenStack for Kubernetes documentation: Ceph operations - Create a Ceph performance test request.

KaaSCephOperationRequest CR perftest specification¶

TechPreview

This section was moved to Mirantis OpenStack for Kubernetes documentation: Ceph operations - KaaSCephOperationRequest CR perftest specification.

KaaSCephOperationRequest perftest status¶

TechPreview

This section was moved to Mirantis OpenStack for Kubernetes documentation: Ceph operations - KaaSCephOperationRequest perftest status.

Delete a managed cluster¶

This section was moved to MOSK documentation: General operations - Delete a managed cluster.

Day-2 operations¶

TechPreview since 2.26.0 (17.1.0 and 16.1.0)

The subsections of this section were moved to MOSK documentation: Host operating system configuration - Day-2 operations.

Day-2 operations workflow¶

TechPreview since 2.26.0 (17.1.0 and 16.1.0)

This section was moved to MOSK documentation: Day-2 operations - Day-2 operations workflow.

Global recommendations for implementation of custom modules¶

This section was moved to MOSK documentation: Day-2 operations - Global recommendations for implementation of custom modules.

Format and structure of a module package¶

TechPreview since 2.26.0 (17.1.0 and 16.1.0)

This section was moved to MOSK documentation: Day-2 operations - Format and structure of a module package.

Modules provided by Container Cloud¶

TechPreview since 2.27.0 (17.2.0 and 16.2.0)

The subsections of this section were moved to mosk-host-os-modules documentation.

irqbalance module¶

TechPreview since 2.27.0 (17.2.0 and 16.2.0)

This section was moved to mosk-host-os-modules documentation: irqbalance module.

package module¶

TechPreview since 2.27.0 (17.2.0 and 16.2.0)

This section was moved to mosk-host-os-modules documentation: package module.

sysctl module¶

TechPreview since 2.27.0 (17.2.0 and 16.2.0)

This section was moved to mosk-host-os-modules documentation: sysctl module.

HostOSConfiguration and HostOSConfigurationModules concepts¶

TechPreview since 2.26.0 (17.1.0 and 16.1.0)

This section was moved to MOSK documentation: Day-2 operations - HostOSConfiguration and HostOSConfigurationModules concepts.

Internal API for day-2 operations¶

TechPreview since 2.26.0 (17.1.0 and 16.1.0)

This section was moved to MOSK documentation: Day-2 operations - Internal API for day-2 operations.

Add a custom module to a Container Cloud deployment¶

TechPreview since 2.26.0 (17.1.0 and 16.1.0)

This section was moved to MOSK documentation: Day-2 operations - Add a custom module to a Container Cloud deployment.

Test a custom or Container Cloud module after creation¶

TechPreview since 2.26.0 (17.1.0 and 16.1.0)

This section was moved to MOSK documentation: Day-2 operations - Test a custom or Container Cloud module after creation.

Retrigger a module configuration¶

This section was moved to MOSK documentation: Day-2 operations - Retrigger a module configuration.

Troubleshooting¶

This section was moved to MOSK documentation: Day-2 operations - Troubleshooting.

Add or update a CA certificate for a MITM proxy using API¶

This section was moved to MOSK documentation: Underlay Kubernetes operations - Add or update a CA certificate for a MITM proxy using API.

Change a cluster configuration¶

This section was moved to MOSK documentation: General Operations - Change a cluster configuration.

Disable a machine¶

TechPreview since 2.25.0 (17.0.0 and 16.0.0) for workers on managed clusters

This section was moved to MOSK documentation: General Operations - Disable a machine.

Configure the parallel update of worker nodes¶

Available since 17.0.0, 16.0.0, and 14.1.0 as GA Available since 14.0.1(0) and 15.0.1 as TechPreview

This section was moved to MOSK documentation: Cluster update - Configure the parallel update of worker nodes.

Create update groups for worker machines¶

Available since 2.27.0 (17.2.0 and 16.2.0)

This section was moved to MOSK documentation: Cluster update - Create update groups for worker machines.

Change the upgrade order of a machine or machine pool¶

This section was moved to MOSK documentation: Cluster update - Change the upgrade order of a machine or machine pool.

Update a managed cluster¶

The subsections of this section were moved to MOSK documentation: Cluster update.

Verify the Container Cloud status before managed cluster update¶

This section was moved to MOSK documentation: Cluster update - Verify the management cluster status before MOSK update.

Update a managed cluster using the Container Cloud web UI¶

This section was moved to MOSK documentation: Cluster update - Update to a major version.

Granularly update a managed cluster using the ClusterUpdatePlan object¶

Available since 2.27.0 (17.2.0 and 16.2.0) TechPreview

This section was moved to MOSK documentation: Cluster update - Granularly update a managed cluster using the ClusterUpdatePlan object.

Update a patch Cluster release of a managed cluster¶

Available since 2.23.2

This section was moved to MOSK documentation: Cluster update - Update a patch Cluster release of a managed cluster.

See also

Delete a cluster machine

Add a Container Cloud cluster to Lens¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Getting access - Add a Container Cloud cluster to Lens.

Connect to the Mirantis Kubernetes Engine web UI¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Getting access.

Connect to a Mirantis Container Cloud cluster¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Getting access - Connect to a MOSK cluster.

Inspect the history of a cluster and machine deployment or update¶

Available since 2.22.0

This section was moved to MOSK Troubleshooting Guide: Inspect the history of a cluster and machine deployment or update.

Operate management clusters¶

The subsections of this section were moved to MOSK documentation: Management cluster operations.

Workflow and configuration of management cluster upgrade¶

This section was moved to MOSK documentation: Management cluster operations - Workflow and configuration of management cluster upgrade.

Schedule Mirantis Container Cloud updates¶

This section was moved to MOSK documentation: Management cluster operations - Schedule Mirantis Container Cloud updates.

Renew the Container Cloud and MKE licenses¶

This section was moved to MOSK documentation: Management cluster operations - Renew the Container Cloud and MKE licenses.

Configure NTP server¶

This section was moved to MOSK documentation: Management cluster operations - Configure NTP server.

Automatically propagate Salesforce configuration to all clusters¶

This section was moved to MOSK documentation: Management cluster operations - Automatically propagate Salesforce configuration to all clusters.

Update the Keycloak IP address on bare metal clusters¶

This section was moved to MOSK documentation: Management cluster operations - Update the Keycloak IP address on bare metal clusters.

Configure host names for cluster machines¶

TechPreview Available since 2.24.0

This section was moved to MOSK documentation: Management cluster operations - Configure host names for cluster machines.

Back up MariaDB on a management cluster¶

The subsections of this section were moved to MOSK documentation: Management cluster operations - Back up MariaDB on a management cluster.

Configure periodic backups of MariaDB¶

This section was moved to MOSK documentation: Management cluster operations - Configure periodic backups of MariaDB.

Verify operability of the MariaDB backup jobs¶

This section was moved to MOSK documentation: Management cluster operations - Verify operability of the MariaDB backup jobs.

Restore MariaDB databases¶

This section was moved to MOSK documentation: Management cluster operations - Restore MariaDB databases.

Change the storage node for MariaDB on bare metal clusters¶

This section was moved to MOSK documentation: Management cluster operations - Change the storage node for MariaDB.

Remove a management cluster¶

This section was moved to MOSK documentation: Management cluster operations - Remove a management cluster.

Warm up the Container Cloud cache¶

TechPreview Available since 2.24.0 and 23.2 for MOSK clusters

This section was moved to MOSK documentation: Management cluster operations - Warm up the Container Cloud cache.

Self-diagnostics for management and managed clusters¶

Available since 2.28.0 (17.3.0 and 16.3.0)

The subsections of this section were moved to MOSK Operations Guide: Bare metal operations - Run cluster self-diagnostics.

Trigger self-diagnostics for a management or managed cluster¶

Available since 2.28.0 (17.3.0 and 16.3.0)

This section was moved to MOSK Operations Guide: Bare metal operations - Trigger self-diagnostics for a management or managed cluster.

Self-upgrades of the Diagnostic Controller¶

Available since 2.28.0 (17.3.0 and 16.3.0)

This section was moved to MOSK Operations Guide: Bare metal operations - Self-upgrades of the Diagnostic Controller.

Diagnostic checks for the bare metal provider¶

Available since 2.28.0 (17.3.0 and 16.3.0) Technology Preview

This section was moved to MOSK Operations Guide: Bare metal operations - Diagnostic checks for the bare metal provider.

Increase memory limits for cluster components¶

This section was moved to MOSK documentation: Underlay Kubernetes operations - Increase memory limits for cluster components.

Set the MTU size for Calico¶

TechPreview Available since 2.24.0 and 2.24.2 for MOSK 23.2

This section was moved to MOSK documentation: Underlay Kubernetes operations - Set the MTU size for Calico.

Increase storage quota for etcd¶

Available since Cluster releases 15.0.3 and 14.0.3

This section was moved to MOSK documentation: Underlay Kubernetes operations - Increase storage quota for etcd.

Configure Kubernetes auditing and profiling¶

Available since 2.24.3 (Cluster releases 15.0.2 and 14.0.2)

This section was moved to MOSK documentation: Underlay Kubernetes operations - Configure Kubernetes auditing and profiling.

Configure TLS certificates for cluster applications¶

Technology Preview

This section was moved to MOSK documentation: Underlay Kubernetes operations - Configure TLS certificates for cluster applications.

Define a custom CA certificate for a private Docker registry¶

This section was moved to MOSK documentation: Underlay Kubernetes operations - Define a custom CA certificate for a private Docker registry.

Enable cluster and machine maintenance mode¶

The subsections of this section were moved to MOSK documentation: General Operations - Enable cluster and machine maintenance mode.

Enable maintenance mode on a cluster and machine using web UI¶

This section was moved to MOSK documentation: General Operations - Enable maintenance mode on a cluster and machine using web UI.

Enable maintenance mode on a cluster and machine using CLI¶

This section was moved to MOSK documentation: General Operations - Enable maintenance mode on a cluster and machine using CLI.

Perform a graceful reboot of a cluster¶

Available since 2.23.0

This section was moved to MOSK documentation: General Operations - Perform a graceful reboot of a cluster.

Delete a cluster machine¶

The subsections of this section were moved to MOSK documentation: Delete a cluster machine.

Precautions for a cluster machine deletion¶

This section was moved to MOSK documentation: Precautions for a cluster machine deletion.

Delete a cluster machine using web UI¶

This section was moved to MOSK documentation: Delete a cluster machine using web UI.

Delete a cluster machine using CLI¶

This section was moved to MOSK documentation: Delete a cluster machine using CLI.

See also

Disable a machine

Manage IAM¶

The subsections of this section were moved to MOSK documentation: IAM operations.

Manage user roles through Container Cloud API¶

The subsections of this section were moved to MOSK documentation: IAM operations - Manage user roles through Container Cloud API.

Available IAM roles and use cases¶

This section was moved to MOSK documentation: Manage user roles through Container Cloud API - Available IAM roles and use cases objects.

Mapping of Keycloak roles to IAM*RoleBinding objects¶

This section was moved to MOSK documentation: Manage user roles through Container Cloud API - Mapping of Keycloak roles to IAM*RoleBinding objects.

Manage user roles through the Container Cloud web UI¶

This section was moved to MOSK documentation: IAM operations - Manage user roles through the Container Cloud web UI.

Manage user roles through Keycloak¶

The subsections of this section were moved to MOSK documentation: IAM operations - Manage user roles through Keycloak.

Container Cloud roles and scopes¶

This section was moved to MOSK documentation: IAM operations - Container Cloud roles and scopes.

Use cases¶

This section was moved to MOSK documentation: Manage user roles through Keycloak - Use cases.

Access the Keycloak Admin Console¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Getting access - Access the Keycloak Admin Console.

Change passwords for IAM users¶

This section was moved to MOSK documentation: IAM operations - Change passwords for IAM users.

Obtain MariaDB credentials for IAM¶

Available since Container Cloud 2.22.0

This section was moved to MOSK documentation: IAM operations - Obtain MariaDB credentials for IAM.

Manage Keycloak truststore using the Container Cloud web UI¶

Available since 2.26.0 (17.1.0 and 16.1.0)

This section was moved to MOSK documentation: IAM operations - Manage Keycloak truststore using the Container Cloud web UI.

Manage StackLight¶

The subsections of this section were moved to MOSK Operations Guide: StackLight operations.

Access StackLight web UIs¶

This section was moved to Mirantis OpenStack for Kubernetes documentation: Getting access.

StackLight logging indices¶

Available since 2.26.0 (17.1.0 and 16.1.0)

This section was moved to MOSK Reference Architecture: StackLight logging indices.

OpenSearch Dashboards¶

The subsections of this section were moved to MOSK Operations Guide: OpenSearch Dashboards.

View OpenSearch Dashboards¶

This section was moved to MOSK Operations Guide: View OpenSearch Dashboards.

Search in OpenSearch Dashboards¶

This section was moved to MOSK Operations Guide: Search in OpenSearch Dashboards.

Export logs from OpenSearch Dashboards to CSV¶

Available since 2.23.0 (12.7.0 and 11.7.0)

This section was moved to MOSK Operations Guide: Export logs from OpenSearch Dashboards to CSV.

Tune OpenSearch performance for the bare metal provider¶

This section was moved to MOSK Operations Guide: Tune OpenSearch performance.

View Grafana dashboards¶

This section was moved to MOSK Operations Guide: View Grafana dashboards.

Export data from Table panels of Grafana dashboards to CSV¶

This section was moved to MOSK Operations Guide: Export data from Table panels of Grafana dashboards to CSV.

Available StackLight alerts¶

The subsections of this section were moved to MOSK Operations Guide: StackLight alerts.

Alert dependencies¶

This section was moved to MOSK Operations Guide: Alert dependencies.

Alertmanager¶

This section was moved to MOSK Operations Guide: StackLight alerts - Alertmanager.

Bond interface¶

Available since 2.24.0 and 2.24.2 for MOSK 23.2

This section was moved to MOSK Operations Guide: Bare metal alerts - Bond interface.

cAdvisor¶

This section was moved to MOSK Operations Guide: StackLight alerts - cAdvisor.

Calico¶

This section was moved to MOSK Operations Guide: Generic alerts - Calico.

Ceph¶

This section was moved to MOSK Operations Guide: StackLight alerts - Ceph.

Docker Swarm¶

This section was moved to MOSK Operations Guide: StackLight alerts - Mirantis Kubernetes Engine.

Elasticsearch Exporter¶

This section was moved to MOSK Operations Guide: StackLight alerts - Elasticsearch Exporter.

Etcd¶

This section was moved to MOSK Operations Guide: Generic alerts - Etcd.

External endpoint¶

This section was moved to MOSK Operations Guide: StackLight alerts - Monitoring of external endpoints.

Fluentd¶

This section was moved to MOSK Operations Guide: StackLight alerts - Fluentd.

General alerts¶

This section was moved to MOSK Operations Guide: General StackLight alerts.

General node alerts¶

This section was moved to MOSK Operations Guide: StackLight alerts - Node.

Grafana¶

This section was moved to MOSK Operations Guide: StackLight alerts - Grafana.

Helm Controller¶

This section was moved to MOSK Operations Guide: Container Cloud alerts - Helm Controller.

Host Operating System Modules Controller¶

TechPreview since 2.28.0 (17.3.0 and 16.3.0)

This section was moved to MOSK Operations Guide: Bare metal alerts - Host Operating System Modules Controller.

Ironic¶

This section was moved to MOSK Operations Guide: Bare metal alerts - Ironic.

Kernel¶

This section was moved to MOSK Operations Guide: Bare metal alerts - Kernel.

Kubernetes applications¶

This section was moved to MOSK Operations Guide: Kubernetes alerts - Kubernetes applications.

Kubernetes resources¶

This section was moved to MOSK Operations Guide: Kubernetes alerts - Kubernetes resources.

Kubernetes storage¶

This section was moved to MOSK Operations Guide: Kubernetes alerts - Kubernetes storage.

Kubernetes system¶

This section was moved to MOSK Operations Guide: Kubernetes alerts - Kubernetes system.

Mirantis Container Cloud¶

This section was moved to MOSK Operations Guide: StackLight alerts - Mirantis Container Cloud.

Mirantis Container Cloud cache¶

This section was moved to MOSK Operations Guide: StackLight alerts - Mirantis Container Cloud cache.

Mirantis Container Cloud controllers¶

Available since Cluster releases 12.7.0 and 11.7.0

This section was moved to MOSK Operations Guide: StackLight alerts - Mirantis Container Cloud controllers.

Mirantis Container Cloud providers¶

Available since Cluster releases 12.7.0 and 11.7.0

This section was moved to MOSK Operations Guide: StackLight alerts - Mirantis Container Cloud providers.

Mirantis Kubernetes Engine¶

This section was moved to MOSK Operations Guide: StackLight alerts - Mirantis Kubernetes Engine.

Node network¶

This section was moved to MOSK Operations Guide: StackLight alerts - Node network.

Node time¶

This section was moved to MOSK Operations Guide: StackLight alerts - Node time.

OpenSearch¶

This section was moved to MOSK Operations Guide: StackLight alerts - OpenSearch.

PostgreSQL¶

This section was moved to MOSK Operations Guide: StackLight alerts - PostgreSQL.

Prometheus¶

This section was moved to MOSK Operations Guide: StackLight alerts - Prometheus.

Prometheus MS Teams¶

This section was moved to MOSK Operations Guide: StackLight alerts - Prometheus MS Teams.

Prometheus Relay¶

This section was moved to MOSK Operations Guide: StackLight alerts - Prometheus Relay.

Release Controller¶

This section was moved to MOSK Operations Guide: StackLight alerts - Release Controller.

ServiceNow¶

This section was moved to MOSK Operations Guide: StackLight alerts - ServiceNow.

Salesforce notifier¶

This section was moved to MOSK Operations Guide: StackLight alerts - Salesforce notifier.

SSL certificates¶

This section was moved to MOSK Operations Guide - StackLight alerts: Monitoring of external endpoints and Container Cloud SSL.

Telegraf¶

This section was moved to MOSK Operations Guide: StackLight alerts - Telegraf.

Telemeter¶

This section was moved to MOSK Operations Guide: StackLight alerts - Telemeter.

Troubleshoot alerts¶

The subsections of this section were moved to MOSK Troubleshooting Guide: Troubleshoot StackLight - Troubleshoot alerts.

Troubleshoot cAdvisor alerts¶

This section was moved to MOSK Troubleshooting Guide: Troubleshoot StackLight - Troubleshoot cAdvisor alerts.

Troubleshoot Helm Controller alerts¶

This section was moved to MOSK Troubleshooting Guide: Troubleshoot StackLight - Troubleshoot Helm Controller alerts.

Troubleshoot Host Operating System Modules Controller alerts¶

This section was moved to MOSK Troubleshooting Guide: Troubleshoot StackLight - Troubleshoot Host Operating System Modules Controller alerts.

Troubleshoot Ubuntu kernel alerts¶

This section was moved to MOSK Troubleshooting Guide: Troubleshoot StackLight - Troubleshoot Ubuntu kernel alerts.

Troubleshoot Kubernetes applications alerts¶

This section was moved to MOSK Troubleshooting Guide: Troubleshoot StackLight - Troubleshoot Kubernetes applications alerts.

Troubleshoot Kubernetes resources alerts¶

This section was moved to MOSK Troubleshooting Guide: Troubleshoot StackLight - Troubleshoot Kubernetes resources alerts.

Troubleshoot Kubernetes storage alerts¶

This section was moved to MOSK Troubleshooting Guide: Troubleshoot StackLight - Troubleshoot Kubernetes storage alerts.

Troubleshoot Kubernetes system alerts¶

This section was moved to MOSK Troubleshooting Guide: Troubleshoot StackLight - Troubleshoot Kubernetes system alerts.

Troubleshoot Mirantis Container Cloud Exporter alerts¶

This section was moved to MOSK Troubleshooting Guide: Troubleshoot StackLight - Troubleshoot Mirantis Container Cloud Exporter alerts.

Troubleshoot Mirantis Kubernetes Engine alerts¶

This section was moved to MOSK Troubleshooting Guide: Troubleshoot StackLight - Troubleshoot Mirantis Kubernetes Engine alerts.

Troubleshoot OpenSearch alerts¶

Available since 2.26.0 (17.1.0 and 16.1.0)

This section was moved to MOSK Troubleshooting Guide: Troubleshoot StackLight - Troubleshoot OpenSearch alerts.

Troubleshoot Release Controller alerts¶

This section was moved to MOSK Troubleshooting Guide: Troubleshoot StackLight - Troubleshoot Release Controller alerts.

Troubleshoot Telemeter client alerts¶

This section was moved to MOSK Troubleshooting Guide: Troubleshoot StackLight - Troubleshoot Telemeter client alerts.

Silence alerts¶

This section was moved to MOSK documentation: Silence alerts.

StackLight rules for Kubernetes network policies¶

Available since Cluster releases 17.0.1 and 16.0.1

This section was moved to MOSK documentation: StackLight rules for Kubernetes network policies.

Configure StackLight¶

The subsections of this section were moved to MOSK Operations Guide: Configure StackLight.

StackLight configuration procedure¶

Thhis section was moved to MOSK Operations Guide: Configure StackLight - StackLight configuration procedure.

StackLight configuration parameters¶

Thhis section was moved to MOSK Operations Guide: Configure StackLight - StackLight configuration parameters.

Verify StackLight after configuration¶

Thhis section was moved to MOSK Operations Guide: Configure StackLight - Verify StackLight after configuration.

Tune StackLight for long-term log retention¶

Available since 2.24.0 and 2.24.2 for MOSK 23.2

This section was moved to MOSK Operations Guide: StackLight operations - Tune StackLight for long-term log retention.

Enable log forwarding to external destinations¶

Available since 2.23.0 and 2.23.1 for MOSK 23.1

This section was moved to MOSK Operations Guide: StackLight operations - Enable log forwarding to external destinations.

Enable remote logging to syslog¶

Deprecated since 2.23.0

This section was moved to MOSK Operations Guide: StackLight operations - Enable remote logging to syslog.

Create logs-based metrics¶

This section was moved to MOSK Operations Guide: StackLight operations - Create logs-based metrics.

Enable generic metric scraping¶

This section was moved to MOSK Operations Guide: StackLight operations - Enable generic metric scraping.

Manage metrics filtering¶

Available since 2.24.0 and 2.24.2 for MOSK 23.2

This section was moved to MOSK Operations Guide: StackLight operations - Manage metrics filtering.

Use S.M.A.R.T. metrics for creating alert rules on bare metal clusters¶

Available since 2.27.0 (Cluster releases 17.2.0 and 16.2.0)

This section was moved to MOSK Operations Guide: StackLight operations - Use S.M.A.R.T. metrics for creating alert rules.

Deschedule StackLight Pods from a worker machine¶

This section was moved to MOSK Operations Guide: Deschedule StackLight Pods from a worker machine.

Calculate the storage retention time¶

Obsolete since 2.26.0 (17.1.0, 16.1.0) for OpenSearch Available since 2.22.0 and 2.23.1 (12.7.0, 11.6.0)

This section was moved to MOSK documentation: Calculate the storage retention time.

Troubleshooting¶

This section was moved to MOSK documentation: Troubleshooting Guide.

Collect cluster logs¶

This section was moved to MOSK documentation: Collect cluster logs.

Cluster deletion or detachment freezes¶

This section was moved to MOSK documentation: Cluster deletion or detachment freezes.

Keycloak admin console becomes inaccessible after changing the theme¶

This section was moved to MOSK documentation: Keycloak admin console becomes inaccessible after changing the theme.

The ‘database space exceeded’ error on large clusters¶

This section was moved to MOSK documentation: The ‘database space exceeded’ error on large clusters.

The auditd events cause ‘backlog limit exceeded’ messages¶

This section was moved to MOSK documentation: The auditd events cause ‘backlog limit exceeded’ messages.

Troubleshoot baremetal-based clusters¶

This section was moved to MOSK documentation: Troubleshoot bare metal.

Log in to the IPA virtual console for hardware troubleshooting¶

This section was moved to MOSK documentation: Log in to the IPA virtual console for hardware troubleshooting.

Bare metal hosts in ‘provisioned registration error’ state after update¶

This section was moved to MOSK documentation: Bare metal hosts in provisioned registration error state after update.

Troubleshoot an operating system upgrade with host restart¶

This section was moved to MOSK documentation: Troubleshoot an operating system upgrade with host restart.

Troubleshoot iPXE boot issues¶

This section was moved to MOSK documentation: Troubleshoot iPXE boot issues.

Provisioning failure due to device naming issues in a bare metal host profile¶

This section was moved to MOSK documentation: Provisioning failure due to device naming issues in a bare metal host profile.

Troubleshoot Ceph¶

This section was moved to MOSK documentation: Troubleshoot Ceph.

Ceph disaster recovery¶

This section was moved to MOSK documentation: Ceph disaster recovery.

Ceph Monitors recovery¶

This section was moved to MOSK documentation: Ceph Monitors recovery.

Remove Ceph OSD manually¶

This section was moved to MOSK documentation: Remove Ceph OSD manually.

KaaSCephOperationRequest failure with a timeout during rebalance¶

This section was moved to MOSK documentation: KaaSCephOperationRequest failure with a timeout during rebalance.

Ceph Monitors store.db size rapidly growing¶

This section was moved to MOSK documentation: Ceph Monitors store.db size rapidly growing.

Replaced Ceph OSD fails to start on authorization¶

This section was moved to MOSK documentation: Replaced Ceph OSD fails to start on authorization.

The ceph-exporter pods are present in the Ceph crash list¶

This section was moved to MOSK documentation: The ceph-exporter pods are present in the Ceph crash list.

Troubleshoot StackLight¶

This section was moved to MOSK documentation: Troubleshoot StackLight.

Patroni replication lag¶

This section was moved to MOSK documentation: Patroni replication lag.

Alertmanager does not send resolve notifications for custom alerts¶

This section was moved to MOSK documentation: Alertmanager does not send resolve notifications for custom alerts.

OpenSearchPVCMismatch alert raises due to the OpenSearch PVC size mismatch¶

This section was moved to MOSK documentation: OpenSearchPVCMismatch alert raises due to the OpenSearch PVC size mismatch.

OpenSearch cluster deadlock due to the corrupted index¶

This section was moved to MOSK documentation: OpenSearch cluster deadlock due to the corrupted index.

Failure of shard relocation in the OpenSearch cluster¶

This section was moved to MOSK documentation: Failure of shard relocation in the OpenSearch cluster.

StackLight pods get stuck with the ‘NodeAffinity failed’ error¶

This section was moved to MOSK documentation: StackLight pods get stuck with the NodeAffinity failed error.

No logs are forwarded to Splunk¶

This section was moved to MOSK documentation: No logs are forwarded to Splunk.

Security Guide¶

This guide is now part of MOSK documentation: Security Guide.

Firewall configuration¶

This section was moved to MOSK documentation: Firewall configuration.

Container Cloud¶

This section was moved to MOSK documentation: Firewall configuration - Container Cloud.

Mirantis Kubernetes Engine¶

For available Mirantis Kubernetes Engine (MKE) ports, refer to MKE Documentation: Open ports to incoming traffic.

StackLight¶

This section was moved to MOSK documentation: Firewall configuration - StackLight.

Ceph¶

This section was moved to MOSK documentation: Firewall configuration - Ceph.

Container images signing and validation¶

Available since 2.26.0 (17.1.0 and 16.1.0) Technology Preview

This section was moved to MOSK documentation: Container images signing and validation.

API Reference¶

This API Reference is now part of MOSK documentation: API Reference.

Public key resources¶

This section was moved to MOSK API Reference: PublicKey resource.

License resource¶

This section was moved to MOSK API Reference: License resource.

Diagnostic resource¶

Available since 2.28.0 (17.3.0 and 16.3.0)

This section was moved to MOSK API Reference: Diagnostic resource.

IAM resources¶

This section was moved to MOSK API Reference: IAM resources resource.

UpdateGroup resource¶

Available since 2.27.0 (17.2.0 and 16.2.0)

This section was moved to MOSK API Reference: UpdateGroup resource.

MCCUpgrade resource¶

This section was moved to MOSK API Reference: MCCUpgrade resource.

ClusterUpdatePlan resource¶

Available since 2.27.0 (17.2.0 and 16.2.0) TechPreview

This section was moved to MOSK API Reference: ClusterUpdatePlan resource.

UpdateAutoPause resource¶

Available since 2.29.0 (17.4.0) Technology Preview

This section was moved to MOSK API Reference: UpdateAutoPause resource.

CacheWarmupRequest resource¶

TechPreview Available since 2.24.0 and 23.2 for MOSK clusters

This section was moved to MOSK API Reference: CacheWarmupRequest resource.

GracefulRebootRequest resource¶

Available since 2.23.0 and 2.23.1 for MOSK 23.1

This section was moved to MOSK API Reference: GracefulRebootRequest resource.

ContainerRegistry resource¶

This section was moved to MOSK API Reference: ContainerRegistry resource.

TLSConfig resource¶

This section was moved to MOSK API Reference: TLSConfig resource.

Bare metal resources¶

The subsections of this section were moved to MOSK API Reference: Cluster resources, Networking resources, and Host OS configuration resources.

BareMetalHost¶

Private API since Container Cloud 2.29.0 (Cluster release 16.4.0)

This section was moved to MOSK API Reference: BareMetalHost resource.

BareMetalHostCredential¶

Available since 2.21.0 and 2.21.1 for MOSK 22.5

This section was moved to MOSK API Reference: BareMetalHostCredential resource.

BareMetalHostInventory¶

Available since Container Cloud 2.29.0 (Cluster release 16.4.0)

This section was moved to MOSK API Reference: BareMetalHostInventory resource.

BareMetalHostProfile¶

This section was moved to MOSK API Reference: BareMetalHostProfile resource.

Cluster¶

This section was moved to MOSK API Reference: Cluster resource.

HostOSConfiguration¶

TechPreview since 2.26.0 (17.1.0 and 16.1.0)

This section was moved to MOSK API Reference: HostOSConfiguration resource.

HostOSConfigurationModules¶

TechPreview since 2.26.0 (17.1.0 and 16.1.0)

This section was moved to MOSK API Reference: HostOSConfigurationModules resource.

IPaddr¶

This section was moved to MOSK API Reference: IPaddr resource.

IpamHost¶

This section was moved to MOSK API Reference: IpamHost resource.

L2Template¶

This section was moved to MOSK API Reference: L2Template resource.

Machine¶

This section was moved to MOSK API Reference: Machine resource.

MetalLBConfig¶

TechPreview since 2.21.0 and 2.21.1 for MOSK 22.5 GA since 2.24.0 for management and regional clusters GA since 2.25.0 for managed clusters

This section was moved to MOSK API Reference: MetalLBConfig resource.

MetalLBConfigTemplate¶

Unsupported since 2.28.0 (17.3.0 and 16.3.0)

This section was moved to MOSK API Reference: MetalLBConfigTemplate resource.

MultiRackCluster¶

TechPreview Available since 2.24.4

This section was moved to MOSK API Reference: MultiRackCluster resource.

Rack¶

TechPreview Available since 2.24.4

This section was moved to MOSK API Reference: Rack resource.

Subnet¶

This section was moved to MOSK API Reference: Subnet resource.

SubnetPool¶

Unsupported since 2.28.0 (17.3.0 and 16.3.0)

This section was moved to MOSK API Reference: SubnetPool resource.

Release Compatibility Matrix¶

The Mirantis Container Cloud Release Compatibility Matrix outlines the specific operating environments that are validated and supported.

The document provides the deployment compatibility for each product release and determines the upgrade paths between major components versions when upgrading. The document also provides the Container Cloud browser compatibility.

A Container Cloud management cluster upgrades automatically when a new product release becomes available. Once the management cluster has been updated, the user may trigger the managed clusters upgrade through the Container Cloud web UI or API.

To view the full components list with their respective versions for each Container Cloud release, refer to the Container Cloud Release Notes related to the release version of your deployment or use the Releases section in the web UI or API.

Caution

The document applies to the Container Cloud regular deployments. For supported configurations of existing Mirantis Kubernetes Engine (MKE) clusters that are not deployed by Container Cloud, refer to MKE Compatibility Matrix.

Compatibility matrix of component versions¶

The following tables outline the compatibility matrices of the most recent major Container Cloud and Cluster releases along with patch releases and their component versions. For details about unsupported releases, see Releases summary.

Major and patch versions update path

The primary distinction between major and patch product versions lies in the fact that major release versions introduce new functionalities, whereas patch release versions predominantly offer minor product enhancements, mostly CVE resolutions for your clusters.

Depending on your deployment needs, you can either update only between major Cluster releases or apply patch updates between major releases. Choosing the latter option ensures you receive security fixes as soon as they become available. Though, be prepared to update your cluster frequently, approximately once every three weeks. Otherwise, you can update only between major Cluster releases as each subsequent major Cluster release includes patch Cluster release updates of the previous major Cluster release.

Legend

Symbol	Definition
	Cluster release is not included in the Container Cloud release yet.
	Latest supported Cluster release to use for cluster deployment or update.
	Deprecated Cluster release that you must update to the latest supported Cluster release. The deprecated Cluster release will become unsupported in one of the following Container Cloud releases. Greenfield deployments based on a deprecated Cluster release are not supported. Use the latest supported Cluster release instead.
	Unsupported Cluster release that blocks automatic upgrade of a management cluster. Update the Cluster release to the latest supported one to unblock management cluster upgrade and obtain newest product features and enhancements.
	Component is included in the Container Cloud release.
	Component is available in the Technology Preview scope. Use it only for testing purposes on staging environments.
	Component is unsupported in the Container Cloud release.

The following table outlines the compatibility matrix for the Container Cloud release series 2.29.x.

Container Cloud compatibility matrix 2.29.x¶
Release	Container Cloud	2.29.5 (current)	2.29.4	2.29.3	2.29.2	2.29.1	2.29.0
Release history	Release date	Jul 14, 2025	Jun 16, 2025	May 20, 2025	Apr 22, 2025	Mar 26, 2025	Mar 11, 2025
Major Cluster releases (managed)	17.4.0 + MOSK 25.1 ^{MKE 3.7.19}
	17.3.0 + MOSK 24.3 ^{MKE 3.7.12}
	17.2.0 + MOSK 24.2 ^{MKE 3.7.8}
	16.4.0 ^{MKE 3.7.19}
	16.3.0 ^{MKE 3.7.12}
	16.2.0 ^{MKE 3.7.8}
Patch Cluster releases (managed)	17.3.x + MOSK 24.3.x	17.3.10 17.3.9 17.3.8 17.3.7 17.3.6 17.3.5 17.3.4	17.3.9 17.3.8 17.3.7 17.3.6 17.3.5 17.3.4	17.3.8 17.3.7 17.3.6 17.3.5 17.3.4	17.3.7 17.3.6 17.3.5 17.3.4	17.3.6 17.3.5 17.3.4	17.3.5 17.3.4
	17.2.x + MOSK 24.2.x
	16.4.x	16.4.5 16.4.4 16.4.3 16.4.2 16.4.1	16.4.4 16.4.3 16.4.2 16.4.1	16.4.3 16.4.2 16.4.1	16.4.2 16.4.1	16.4.1
	16.3.x	16.3.10 16.3.9 16.3.8 16.3.7 16.3.6 16.3.5 16.3.4 16.3.3	16.3.9 16.3.8 16.3.7 16.3.6 16.3.5 16.3.4 16.3.3	16.3.8 16.3.7 16.3.6 16.3.5 16.3.4 16.3.3	16.3.7 16.3.6 16.3.5 16.3.4 16.3.3	16.3.6 16.3.5 16.3.4 16.3.3	16.3.5 16.3.4 16.3.3
	16.2.x
Managed cluster	Mirantis Kubernetes Engine (MKE)	3.7.23 ^{17.3.10, 16.4.5, 16.3.10}	3.7.23 ^{17.3.9, 16.4.4, 16.3.9}	3.7.22 ^{17.3.8, 16.4.3, 16.3.8}	3.7.20 ^{17.3.7, 16.4.2, 16.3.7}	3.7.20 ^{17.3.6, 16.4.1, 16.3.6}	3.7.19 ^{17.4.0, 16.4.0}
Container orchestration	Kubernetes	1.27 ^{17.x, 16.x}	1.27 ^{17.x, 16.x}	1.27 ^{17.x, 16.x}	1.27 ^{17.x, 16.x}	1.27 ^{17.x, 16.x}	1.27 ^{17.x, 16.x}
Container runtime	Mirantis Container Runtime (MCR)	25.0.8 12 ^{17.4.0, 16.4.0, 16.4.5} 23.0.15 12 ^{17.3.10, 16.3.10}	25.0.8 12 ^{17.4.0, 16.4.0, 16.4.4} 23.0.15 12 ^{17.3.9, 16.3.9}	25.0.8 12 ^{17.4.0, 16.4.0, 16.4.3} 23.0.15 12 ^{17.3.8, 16.3.8}	25.0.8 11 ^{17.4.0, 16.4.0, 16.4.2} 23.0.15 11 ^{17.3.7, 16.3.7}	25.0.8 11 ^{17.4.0, 16.4.0, 16.4.1} 23.0.15 11 ^{17.3.6, 16.3.6}	25.0.8 ^{17.4.0, 16.4.0} 23.0.15 ^{17.3.5, 16.3.5}
OS distributions	Ubuntu	22.04	22.04	22.04	22.04	22.04	22.04
Infrastructure platform	Bare metal 8	kernel 5.15.0-142-generic ^Jammy	kernel 5.15.0-140-generic ^Jammy	kernel 5.15.0-135-generic ^Jammy	kernel 5.15.0-135-generic ^Jammy	kernel 5.15.0-134-generic ^Jammy	kernel 5.15.0-131-generic ^Jammy
	MOSK Yoga or Antelope with OVS 3
	OpenStack (Octavia) 3 4	Queens Yoga Antelope	Queens Yoga Antelope	Queens Yoga Antelope	Queens Yoga Antelope	Queens Yoga Antelope	Queens Yoga Antelope
Software defined storage	Ceph	18.2.7-2.cve ^17.3.10	18.2.7-2.cve ^17.3.9	18.2.7-1.cve ^17.3.8	18.2.4-13.cve ^17.3.7	18.2.4-13.cve ^17.3.6	18.2.4-12.cve ^17.4.0 18.2.4-11.cve ^17.3.5
Logging, monitoring, and alerting	StackLight

The following drop-down tables outline compatibility matrices for several previous releases of Container Cloud.

Container Cloud compatibility matrix 2.28.x

Release	Container Cloud	2.28.5	2.28.4	2.28.3	2.28.2	2.28.1	2.28.0
Release history	Release date	Feb 03, 2025	Jan 06, 2025	Dec 09, 2024	Nov 18, 2024	Oct 30, 2024	Oct 16, 2024
Major Cluster releases (managed)	17.3.0 + MOSK 24.3 ^{MKE 3.7.12}
	17.2.0 + MOSK 24.2 ^{MKE 3.7.8}
	17.1.0 + MOSK 24.1 ^{MKE 3.7.5}
	16.3.0 ^{MKE 3.7.12}
	16.2.0 ^{MKE 3.7.8}
	16.1.0 ^{MKE 3.7.5}
Patch Cluster releases (managed)	17.3.x + MOSK 24.3.x	17.3.5 17.3.4	17.3.4
	17.2.x + MOSK 24.2.x		17.2.7 17.2.6 17.2.5 17.2.4 17.2.3	17.2.7 17.2.6 17.2.5 17.2.4 17.2.3	17.2.6 17.2.5 17.2.4 17.2.3	17.2.5 17.2.4 17.2.3	17.2.4 17.2.3
	17.1.x + MOSK 24.1.x
	16.3.x	16.3.5 16.3.4 16.3.3 16.3.2 16.3.1	16.3.4 16.3.3 16.3.2 16.3.1	16.3.3 16.3.2 16.3.1	16.3.2 16.3.1	16.3.1
	16.2.x		16.2.7 16.2.6 16.2.5 16.2.4 16.2.3	16.2.7 16.2.6 16.2.5 16.2.4 16.2.3	16.2.6 16.2.5 16.2.4 16.2.3	16.2.5 16.2.4 16.2.3	16.2.4 16.2.3 16.2.2 16.2.1
	16.1.x
Fully managed cluster	Mirantis Kubernetes Engine (MKE)	3.7.18 ^{17.3.5, 16.3.5}	3.7.17 ^{17.3.4, 16.3.4}	3.7.16 ^{17.2.7, 16.3.3, 16.2.7}	3.7.16 ^{17.2.6, 16.3.2, 16.2.6}	3.7.15 ^{17.2.5, 16.3.1, 16.2.5}	3.7.12 ^{17.3.0, 16.3.0}
Container orchestration	Kubernetes	1.27 ^{17.x, 16.x}	1.27 ^{17.x, 16.x}	1.27 ^{17.x, 16.x}	1.27 ^{17.x, 16.x}	1.27 ^{17.x, 16.x}	1.27 ^{17.x, 16.x}
Container runtime	Mirantis Container Runtime (MCR)	23.0.15 ^{17.3.5, 16.3.5} 23.0.14 ^{17.3.0, 16.3.0}	23.0.15 ^{17.3.4, 16.3.4} 23.0.14 ^{17.3.0, 16.3.0}	23.0.14 ^{17.3.0, 16.3.x} 23.0.11 ^{17.2.7, 16.2.7}	23.0.14 ^{17.3.0, 16.3.x} 23.0.11 ^{17.2.6, 16.2.6}	23.0.14 ^{17.3.0, 16.3.x} 23.0.11 ^{17.2.5, 16.2.5}	23.0.14 ^{17.3.0, 16.3.0} 23.0.11 ^{17.2.4, 16.2.4}
OS distributions	Ubuntu	22.04 9	22.04 9	22.04 9	22.04 9	22.04 9	22.04 9
Infrastructure platform	Bare metal 8	kernel 5.15.0-130-generic ^{Jammy, Focal}	kernel 5.15.0-126-generic ^{Jammy, Focal}	kernel 5.15.0-125-generic ^{Jammy, Focal}	kernel 5.15.0-124-generic ^{Jammy, Focal}	kernel 5.15.0-122-generic ^{Jammy, Focal}	kernel 5.15.0-119-generic ^{Jammy, Focal}
	MOSK Yoga or Antelope with OVS 3
	OpenStack (Octavia) 3 4	Queens Yoga Antelope	Queens Yoga Antelope	Queens Yoga Antelope	Queens Yoga Antelope	Queens Yoga Antelope	Queens Yoga Antelope
Software defined storage	Ceph	18.2.4-11.cve ^{17.3.5, 16.3.5}	18.2.4-11.cve ^{17.3.4, 16.3.4}	18.2.4-10.cve ^{17.2.7, 16.3.3, 16.2.7}	18.2.4-8.cve ^{17.2.6, 16.3.2, 16.2.6}	18.2.4-6.cve ^16.3.1 18.2.4-7.cve ^{17.2.5, 16.2.5}	18.2.4-6.cve ^{17.3.0, 16.3.0}
	Rook	1.13.5-28 ^{17.3.5, 16.3.5}	1.13.5-28 ^{17.3.4, 16.3.4}	1.13.5-26 ^{17.2.7, 16.3.3, 16.2.7}	1.13.5-23 ^{17.2.6, 16.3.2, 16.2.6}	1.13.5-21 ^16.3.1 1.13.5-22 ^{17.2.5, 16.2.5}	1.13.5-21 ^{17.3.0, 16.3.0}
Logging, monitoring, and alerting	StackLight

Container Cloud compatibility matrix 2.27.x

Release	Container Cloud	2.27.4	2.27.3	2.27.2	2.27.1	2.27.0
Release history	Release date	Sep 16, 2024	Aug 27, 2024	Aug 05, 2024	July 16, 2024	July 02, 2024
Major Cluster releases (managed)	17.2.0 + MOSK 24.2 ^{MKE 3.7.8}
	17.1.0 + MOSK 24.1 ^{MKE 3.7.5}
	16.2.0 ^{MKE 3.7.8}
	16.1.0 ^{MKE 3.7.5}
Patch Cluster releases (managed)	17.2.x + MOSK 24.2.x	17.2.4 17.2.3	17.2.3
	17.1.x + MOSK 24.1.x	17.1.7+24.1.7 17.1.6+24.1.6 17.1.5+24.1.5	17.1.7+24.1.7 17.1.6+24.1.6 17.1.5+24.1.5	17.1.7+24.1.7 17.1.6+24.1.6 17.1.5+24.1.5	17.1.6+24.1.6 17.1.5+24.1.5	17.1.5+24.1.5
	16.2.x	16.2.4 16.2.3 16.2.2 16.2.1	16.2.3 16.2.2 16.2.1	16.2.2 16.2.1	16.2.1
	16.1.x	16.1.7 16.1.6 16.1.5	16.1.7 16.1.6 16.1.5	16.1.7 16.1.6 16.1.5	16.1.6 16.1.5	16.1.5
Fully managed cluster	Mirantis Kubernetes Engine (MKE)	3.7.12 ^{17.2.4, 16.2.4}	3.7.12 ^{17.2.3, 16.2.3}	3.7.11 ^{17.1.7, 16.2.2, 16.1.7}	3.7.10 ^{17.1.6, 16.2.1, 16.1.6}	3.7.8 ^{17.2.0, 16.2.0}
Attached managed cluster	MKE 7			3.6.8 ^19.1.0 3.6.1 ^19.0.0 3.5.5 ^18.1.0 3.5.3 ^18.0.0	3.6.8 ^19.1.0 3.6.1 ^19.0.0 3.5.5 ^18.1.0 3.5.3 ^18.0.0	3.6.8 ^19.1.0 3.6.1 ^19.0.0 3.5.5 ^18.1.0 3.5.3 ^18.0.0
Container orchestration	Kubernetes	1.27 ^{17.x, 16.x}	1.27 ^{17.x, 16.x}	1.27 ^{17.x, 16.x}	1.27 ^{17.x, 16.x}	1.27 ^{17.x, 16.x}
Container runtime	Mirantis Container Runtime (MCR)	23.0.11 ^{17.2.x, 16.2.x} 10 23.0.9 ^{17.1.x, 16.1.x} 10	23.0.11 ^{17.2.x, 16.2.x} 10 23.0.9 ^{17.1.x, 16.1.x} 10	23.0.11 ^{17.2.x, 16.2.x} 10 23.0.9 ^{17.1.x, 16.1.x} 10	23.0.11 ^{17.2.x, 16.2.x} 10 23.0.9 ^{17.1.x, 16.1.x} 10	23.0.11 ^{17.2.x, 16.2.x}
OS distributions	Ubuntu	22.04 9 20.04	22.04 9 20.04	22.04 9 20.04	22.04 9 20.04	22.04 9 20.04
Infrastructure platform	Bare metal 8	kernel 5.15.0-119-generic ^Jammy kernel 5.15.0-118-generic ^Focal	kernel 5.15.0-117-generic ^{Jammy, Focal}	kernel 5.15.0-116-generic ^Jammy kernel 5.15.0-113-generic ^Focal	kernel 5.15.0-113-generic	kernel 5.15.0-107-generic
	MOSK Yoga or Antelope with OVS 3
	OpenStack (Octavia) 3 4	Queens Yoga Antelope	Queens Yoga Antelope	Queens Yoga Antelope	Queens Yoga Antelope	Queens Yoga Antelope
	VMware vSphere 5			7.0, 6.7	7.0, 6.7	7.0, 6.7
Software defined storage	Ceph	18.2.4-4.cve ^{17.2.4, 16.2.4}	18.2.4-3.cve ^{17.2.3, 16.2.3}	18.2.3-2.cve ^16.2.2 17.2.7-15.cve ^{17.1.7, 16.1.7}	18.2.3-2.cve ^16.2.1 17.2.7-15.cve ^{17.1.6, 16.1.6}	18.2.3-1.release ^{17.2.0, 16.2.0}
	Rook	1.13.5-19 ^{17.2.4, 16.2.4}	1.13.5-18 ^{17.2.3, 16.2.3}	1.13.5-16 ^16.2.2 1.12.10-21 ^{17.1.7, 16.1.7}	1.13.5-16 ^16.2.1 1.12.10-21 ^{17.1.6, 16.1.6}	1.13.5-15 ^{17.2.0, 16.2.0}
Logging, monitoring, and alerting	StackLight

Container Cloud compatibility matrix 2.26.x

Release	Container Cloud	2.26.5	2.26.4	2.26.3	2.26.2	2.26.1	2.26.0
Release history	Release date	June 18, 2024	May 20, 2024	Apr 29, 2024	Apr 08, 2024	Mar 20, 2024	Mar 04, 2024
Major Cluster releases (managed)	17.1.0 + MOSK 24.1 ^{MKE 3.7.5}
	17.0.0 + MOSK 23.3 ^{MKE 3.7.1}
	16.1.0 ^{MKE 3.7.5}
	16.0.0 ^{MKE 3.7.1}
Patch Cluster releases (managed)	17.1.x + MOSK 24.1.x	17.1.5+24.1.5 17.1.4+24.1.4 17.1.3+24.1.3 17.1.2+24.1.2 17.1.1+24.1.1	17.1.4+24.1.4 17.1.3+24.1.3 17.1.2+24.1.2 17.1.1+24.1.1	17.1.3+24.1.3 17.1.2+24.1.2 17.1.1+24.1.1	17.1.2+24.1.2 17.1.1+24.1.1	17.1.1+24.1.1
	17.0.x + MOSK 23.3.x	17.0.4+23.3.4	17.0.4+23.3.4	17.0.4+23.3.4	17.0.4+23.3.4	17.0.4+23.3.4	17.0.4+23.3.4 17.0.3+23.3.3 17.0.2+23.3.2 17.0.1+23.3.1
	16.1.x	16.1.5 16.1.4 16.1.3 16.1.2 16.1.1	16.1.4 16.1.3 16.1.2 16.1.1	16.1.3 16.1.2 16.1.1	16.1.2 16.1.1	16.1.1
	16.0.x	16.0.4	16.0.4	16.0.4	16.0.4	16.0.4	16.0.4 16.0.3 16.0.2 16.0.1
Fully managed cluster	Mirantis Kubernetes Engine (MKE)	3.7.8 ^{17.1.5, 16.1.5}	3.7.8 ^{17.1.4, 16.1.4}	3.7.7 ^{17.1.3, 16.1.3}	3.7.6 ^{17.1.2, 16.1.2}	3.7.5 ^{17.1.1, 16.1.1}	3.7.5 ^{17.1.0, 16.1.0}
Attached managed cluster	MKE 7	3.6.8 ^19.1.0 3.6.1 ^19.0.0 3.5.5 ^18.1.0 3.5.3 ^18.0.0	3.6.8 ^19.1.0 3.6.1 ^19.0.0 3.5.5 ^18.1.0 3.5.3 ^18.0.0	3.6.8 ^19.1.0 3.6.1 ^19.0.0 3.5.5 ^18.1.0 3.5.3 ^18.0.0	3.6.8 ^19.1.0 3.6.1 ^19.0.0 3.5.5 ^18.1.0 3.5.3 ^18.0.0	3.6.8 ^19.1.0 3.6.1 ^19.0.0 3.5.5 ^18.1.0 3.5.3 ^18.0.0	3.6.8 ^19.1.0 3.6.1 ^19.0.0 3.5.5 ^18.1.0 3.5.3 ^18.0.0
Container orchestration	Kubernetes	1.27 ^{17.1.x, 16.1.x}	1.27 ^{17.1.x, 16.1.x}	1.27 ^{17.1.x, 16.1.x}	1.27 ^{17.1.x, 16.1.x}	1.27 ^{17.1.x, 16.1.x}	1.27 ^{17.1.x, 16.1.x}
Container runtime	Mirantis Container Runtime (MCR)	23.0.9 ^{17.1.x, 16.1.x} 2	23.0.9 ^{17.1.x, 16.1.x} 2	23.0.9 ^{17.1.x, 16.1.x} 2	23.0.9 ^{17.1.x, 16.1.x} 2	23.0.9 ^{17.1.x, 16.1.x}	23.0.9 ^{17.1.x, 16.1.x}
OS distributions	Ubuntu	20.04	20.04	20.04	20.04	20.04	20.04
Infrastructure platform	Bare metal 8	kernel 5.15.0-107-generic	kernel 5.15.0-105-generic	kernel 5.15.0-102-generic	kernel 5.15.0-101-generic	kernel 5.15.0-97-generic	kernel 5.15.0-92-generic
	MOSK Yoga or Antelope with OVS 3
	OpenStack (Octavia) 3 4	Queens Yoga Antelope	Queens Yoga Antelope	Queens Yoga Antelope	Queens Yoga Antelope	Queens Yoga Antelope	Queens Yoga Antelope
	VMware vSphere 5	7.0, 6.7	7.0, 6.7	7.0, 6.7	7.0, 6.7	7.0, 6.7	7.0, 6.7
Software defined storage	Ceph	17.2.7-13.cve ^{17.1.5, 16.1.5}	17.2.7-12.cve ^{17.1.4, 16.1.4}	17.2.7-11.cve ^{17.1.3, 16.1.3}	17.2.7-10.release ^{17.1.2, 16.1.2}	17.2.7-9.release ^{17.1.1, 16.1.1}	17.2.7-8.release ^{17.1.0, 16.1.0}
	Rook	1.12.10-19 ^{17.1.5, 16.1.5}	1.12.10-18 ^{17.1.4, 16.1.4}	1.12.10-17 ^{17.1.3, 16.1.3}	1.12.10-16 ^{17.1.2, 16.1.2}	1.12.10-14 ^{17.1.1, 16.1.1}	1.12.10-13 ^{17.1.0, 16.1.0}
Logging, monitoring, and alerting	StackLight

Container Cloud compatibility matrix 2.25.x

Release	Container Cloud	2.25.4	2.25.3	2.25.2	2.25.1	2.25.0
Release history	Release date	Jan 10, 2024	Dec 18, 2023	Dec 05, 2023	Nov 27, 2023	Nov 06, 2023
	17.0.0 + MOSK 23.3 ^{MKE 3.7.1}
	16.0.0 ^{MKE 3.7.1}
	15.0.1 + MOSK 23.2 ^{MKE 3.6.5}
	14.1.0 1 ^{MKE 3.6.6}
	14.0.1 ^{MKE 3.6.5}
	12.7.0 + MOSK 23.1 ^{MKE 3.5.7}
	11.7.0 ^{MKE 3.5.7}
Patch Cluster releases (managed)	17.0.x + MOSK 23.3.x	17.0.4+23.3.4 17.0.3+23.3.3 17.0.2+23.3.2 17.0.1+23.3.1	17.0.3+23.3.3 17.0.2+23.3.2 17.0.1+23.3.1	17.0.2+23.3.2 17.0.1+23.3.1	17.0.1+23.3.1
	16.0.x	16.0.4 16.0.3 16.0.2 16.0.1	16.0.3 16.0.2 16.0.1	16.0.2 16.0.1	16.0.1
	15.0.x + MOSK 23.2.x	15.0.4+23.2.3	15.0.4+23.2.3	15.0.4+23.2.3	15.0.4+23.2.3	15.0.4+23.2.3
	14.0.x	14.0.4	14.0.4	14.0.4	14.0.4	14.0.4
Fully managed cluster	Mirantis Kubernetes Engine (MKE)	3.7.3 ^{Since 17.0.3, 16.0.3} 3.7.2 ^{Since 17.0.1, 16.0.1} 3.7.1 ^{17.0.0, 16.0.0}	3.7.3 ^{Since 17.0.3, 16.0.3} 3.7.2 ^{Since 17.0.1, 16.0.1} 3.7.1 ^{17.0.0, 16.0.0}	3.7.2 ^{Since 17.0.1, 16.0.1} 3.7.1 ^{17.0.0, 16.0.0}	3.7.2 ^{Since 17.0.1, 16.0.1} 3.7.1 ^{17.0.0, 16.0.0}	3.7.1 ^{17.0.0, 16.0.0}
Attached managed cluster	MKE 7	3.6.8 ^19.1.0 3.6.1 ^19.0.0 3.5.5 ^18.1.0 3.5.3 ^18.0.0	3.6.8 ^19.1.0 3.6.1 ^19.0.0 3.5.5 ^18.1.0 3.5.3 ^18.0.0	3.6.8 ^19.1.0 3.6.1 ^19.0.0 3.5.5 ^18.1.0 3.5.3 ^18.0.0
Container orchestration	Kubernetes	1.27 ^{17.0.x, 16.0.x}	1.27 ^{17.0.x, 16.0.x}	1.27 ^{17.0.x, 16.0.x}	1.27 ^{17.0.x, 16.0.x}	1.27 ^{17.0.0, 16.0.0}
Container runtime	Mirantis Container Runtime (MCR)	23.0.7 ^{17.0.x, 16.0.x}	23.0.7 ^{17.0.x, 16.0.x}	23.0.7 ^{17.0.x, 16.0.x}	23.0.7 ^{17.0.x, 16.0.x}	23.0.7 ^{17.0.0, 16.0.0}
OS distributions	Ubuntu	20.04	20.04	20.04	20.04	20.04
Infrastructure platform	Bare metal 8	kernel 5.15.0-86-generic	kernel 5.15.0-86-generic	kernel 5.15.0-86-generic	kernel 5.15.0-86-generic	kernel 5.15.0-86-generic
	MOSK Yoga or Antelope with Tungsten Fabric 3
	MOSK Yoga or Antelope with OVS 3
	OpenStack (Octavia) 3 4	Queens Yoga Antelope	Queens Yoga Antelope	Queens Yoga Antelope	Queens Yoga Antelope	Queens Yoga Antelope
	VMware vSphere 5	7.0, 6.7	7.0, 6.7	7.0, 6.7	7.0, 6.7	7.0, 6.7
Software defined storage	Ceph	17.2.6-8.cve ^{Since 17.0.3, 16.0.3} 17.2.6-5.cve ^{17.0.2, 16.0.2} 17.2.6-2.cve ^{17.0.1, 16.0.1} 17.2.6-cve-1 ^{17.0.0, 16.0.0, 14.1.0}	17.2.6-8.cve ^{17.0.3, 16.0.3} 17.2.6-5.cve ^{17.0.2, 16.0.2} 17.2.6-2.cve ^{17.0.1, 16.0.1} 17.2.6-cve-1 ^{17.0.0, 16.0.0, 14.1.0}	17.2.6-5.cve ^{17.0.2, 16.0.2} 17.2.6-2.cve ^{17.0.1, 16.0.1} 17.2.6-cve-1 ^{17.0.0, 16.0.0, 14.1.0}	17.2.6-2.cve ^{17.0.1, 16.0.1} 17.2.6-cve-1 ^{17.0.0, 16.0.0, 14.1.0}	17.2.6-cve-1 ^{17.0.0, 16.0.0, 14.1.0}
	Rook	1.11.11-22 ^{17.0.4, 16.0.4} 1.11.11-21 ^{17.0.3, 16.0.3} 1.11.11-17 ^{17.0.2, 16.0.2} 1.11.11-15 ^{17.0.1, 16.0.1} 1.11.11-13 ^{17.0.0, 16.0.0, 14.1.0}	1.11.11-21 ^{17.0.3, 16.0.3} 1.11.11-17 ^{17.0.2, 16.0.2} 1.11.11-15 ^{17.0.1, 16.0.1} 1.11.11-13 ^{17.0.0, 16.0.0, 14.1.0}	1.11.11-17 ^{17.0.2, 16.0.2} 1.11.11-15 ^{S17.0.1, 16.0.1} 1.11.11-13 ^{17.0.0, 16.0.0, 14.1.0}	1.11.11-15 ^{17.0.1, 16.0.1} 1.11.11-13 ^{17.0.0, 16.0.0, 14.1.0}	1.11.11-13 ^{17.0.0, 16.0.0, 14.1.0}
Logging, monitoring, and alerting	StackLight

Container Cloud compatibility matrix 2.24.x

Release	Container Cloud	2.24.5	2.24.4	2.24.3	2.24.2	2.24.0 2.24.1 0
Release history	Release date	Sep 26, 2023	Sep 14, 2023	Aug 29, 2023	Aug 21, 2023	Jul 20, 2023 Jul 27, 2023
Major Cluster releases (managed)	15.0.1 + MOSK 23.2 ^{MKE 3.6.5}
	14.0.1 ^{MKE 3.6.5}
	14.0.0 ^{MKE 3.6.5}
	12.7.0 + MOSK 23.1 ^{MKE 3.5.7}
	11.7.0 ^{MKE 3.5.7}
Patch Cluster releases (managed)	15.0.x + MOSK 23.2.x	15.0.4+23.2.3 15.0.3+23.2.2 15.0.2+23.2.1	15.0.3+23.2.2 15.0.2+23.2.1	15.0.2+23.2.1
	14.0.x	14.0.4 14.0.3 14.0.2	14.0.3 14.0.2	14.0.2
Managed cluster	Mirantis Kubernetes Engine (MKE)	3.6.6 ^{Since 15.0.2, 14.0.2} 3.6.5 ^{15.0.1, 14.0.1}	3.6.6 ^{Since 15.0.2, 14.0.2} 3.6.5 ^{15.0.1, 14.0.1}	3.6.6 ^{15.0.2, 14.0.2} 3.6.5 ^{15.0.1, 14.0.1}	3.6.5 ^{15.0.1, 14.0.1}	3.6.5 ^14.0.0
Container orchestration	Kubernetes	1.24 ^{15.0.x, 14.0.x}	1.24 ^{15.0.x, 14.0.x}	1.24 ^{15.0.x, 14.0.x}	1.24 ^{15.0.1, 14.0.1}	1.24 ^14.0.0
Container runtime	Mirantis Container Runtime (MCR)	20.10.17 ^{15.0.x, 14.0.x}	20.10.17 ^{15.0.x, 14.0.x}	20.10.17 2 ^{15.0.x, 14.0.x}	20.10.17 ^{15.0.1, 14.0.1}	20.10.17 ^14.0.0
OS distributions	Ubuntu	20.04	20.04	20.04	20.04	20.04
Infrastructure platform	Bare metal	kernel 5.4.0-150-generic	kernel 5.4.0-150-generic	kernel 5.4.0-150-generic	kernel 5.4.0-150-generic	kernel 5.4.0-150-generic
	MOSK Yoga or Antelope with Tungsten Fabric 3
	MOSK Yoga or Antelope with OVS 3
	OpenStack (Octavia) 3 4	Queens Yoga	Queens Yoga	Queens Yoga	Queens Yoga	Queens Yoga
	VMware vSphere 5	7.0, 6.7	7.0, 6.7	7.0, 6.7	7.0, 6.7	7.0, 6.7
Software defined storage	Ceph 6	17.2.6-cve-1 ^{Since 15.0.2, 14.0.2} 17.2.6-rel-5 ^{15.0.1, 14.0.1}	17.2.6-cve-1 ^{Since 15.0.2, 14.0.2} 17.2.6-rel-5 ^{15.0.1, 14.0.1}	17.2.6-cve-1 ^{15.0.2, 14.0.2} 17.2.6-rel-5 ^{15.0.1, 14.0.1}	17.2.6-rel-5	17.2.6-rel-5 16.2.11-cve-4 16.2.11
	Rook 6	1.11.4-12 ^{Since 15.0.3, 14.0.3} 1.11.4-11 ^{15.0.2, 14.0.2} 1.11.4-10 ^{15.0.1, 14.0.1}	1.11.4-12 ^{15.0.3, 14.0.3} 1.11.4-11 ^{15.0.2, 14.0.2} 1.11.4-10 ^{15.0.1, 14.0.1}	1.11.4-11 ^{15.0.2, 14.0.2} 1.11.4-10 ^{15.0.1, 14.0.1}	1.11.4-10	1.11.4-10 1.10.10-10 1.0.0-20230120144247
Logging, monitoring, and alerting	StackLight

Container Cloud compatibility matrix 2.23.x

Release	Container Cloud	2.23.5	2.23.4	2.23.3	2.23.2	2.23.1	2.23.0
Release history	Release date	Jun 05, 2023	May 22, 2023	May 04, 2023	Apr 20, 2023	Apr 04, 2023	Mar 07, 2023
Major Cluster releases (managed)	12.7.0 + MOSK 23.1 ^{MKE 3.5.7}
	12.5.0 + MOSK 22.5 ^{MKE 3.5.5}
	11.7.0 ^{MKE 3.5.7}
	11.6.0 ^{MKE 3.5.5}
Patch Cluster releases (managed)	12.7.x + MOSK 23.1.x	12.7.4 + 23.1.4 12.7.3 + 23.1.3 12.7.2 + 23.1.2 12.7.1 + 23.1.1	12.7.3 + 23.1.3 12.7.2 + 23.1.2 12.7.1 + 23.1.1	12.7.2 + 23.1.2 12.7.1 + 23.1.1	12.7.1 + 23.1.1
	11.7.x	11.7.4 11.7.3 11.7.2 11.7.1	11.7.3 11.7.2 11.7.1	11.7.2 11.7.1	11.7.1
Managed cluster	Mirantis Kubernetes Engine (MKE)	3.5.7 ^{12.7.x, 11.7.x}	3.5.7 ^{12.7.x, 11.7.x}	3.5.7 ^{12.7.x, 11.7.x}	3.5.7 ^{12.7.x, 11.7.x}	3.5.7 ^{12.7.0, 11.7.0}	3.5.7 ^11.7.0
Container orchestration	Kubernetes	1.21 ^{12.7.x, 11.7.x}	1.21 ^{12.7.x, 11.7.x}	1.21 ^{12.7.x, 11.7.x}	1.21 ^{12.7.x, 11.7.x}	1.21 ^{12.7.0, 11.7.0}	1.21 ^{12.5.0, 11.7.0}
Container runtime	Mirantis Container Runtime (MCR) 2	20.10.13	20.10.13	20.10.13	20.10.13	20.10.13	20.10.13
OS distributions	Ubuntu	20.04	20.04	20.04	20.04	20.04	20.04
Infrastructure platform	Bare metal	kernel 5.4.0-137-generic	kernel 5.4.0-137-generic	kernel 5.4.0-137-generic	kernel 5.4.0-137-generic	kernel 5.4.0-137-generic	kernel 5.4.0-137-generic
	MOSK Victoria or Yoga with Tungsten Fabric 3
	MOSK Victoria or Yoga with OVS 3
	OpenStack (Octavia) 3 4	Queens Victoria Yoga	Queens Victoria Yoga	Queens Victoria Yoga	Queens Victoria Yoga	Queens Victoria Yoga	Queens Victoria Yoga
	VMware vSphere 5	7.0, 6.7	7.0, 6.7	7.0, 6.7	7.0, 6.7	7.0, 6.7	7.0, 6.7
Software defined storage	Ceph 6	16.2.11-cve-4 16.2.11-cve-2 16.2.11	16.2.11-cve-4 16.2.11-cve-2 16.2.11	16.2.11-cve-4 16.2.11-cve-2 16.2.11	16.2.11-cve-2 16.2.11	16.2.11	16.2.11
	Rook 6	1.10.10-10 1.10.10-9 1.0.0-20230120144247	1.10.10-10 1.10.10-9 1.0.0-20230120144247	1.10.10-10 1.10.10-9 1.0.0-20230120144247	1.10.10-9 1.0.0-20230120144247	1.0.0-20230120144247	1.0.0-20230120144247
Logging, monitoring, and alerting	StackLight

0

Container Cloud 2.23.5 or 2.24.0 automatically upgrades to the 2.24.1 patch release containing several hot fixes.

1

The major Cluster release 14.1.0 is dedicated for the vSphere provider only. This is the last Cluster release for the vSphere provider based on MCR 20.10 and MKE 3.6.6 with Kubernetes 1.24.

Container Cloud 2.25.1 introduces the patch Cluster release 16.0.1 that supports the vSphere provider on MCR 23.0.7 and MKE 3.7.2 with Kubernetes 1.27. For details, see External vSphere CCM with CSI supporting vSphere 6.7 on Kubernetes 1.27.

2(1,2,3,4,5,6)

In Container Cloud 2.26.2, docker-ee-cli is updated to 23.0.10 for MCR 23.0.9 to fix several CVEs.
In Container Cloud 2.24.3, docker-ee-cli is updated to 20.10.18 for MCR 20.10.17 to fix the following CVEs: CVE-2023-28840, CVE-2023-28642, CVE-2022-41723.

3(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17)

OpenStack Antelope is supported as TechPreview since MOSK 23.3.
A Container Cloud cluster based on MOSK Yoga or Antelope with Tungsten Fabric is supported as TechPreview since Container Cloud 2.25.1. Since Container Cloud 2.26.0, support for this configuration is suspended. If you still require this configuration, contact Mirantis support for further information.
OpenStack Victoria is supported until September, 2023. MOSK 23.2 is the last release version where OpenStack Victoria packages are updated.

If you have not already upgraded your OpenStack version to Yoga, Mirantis highly recommends doing this during the course of the MOSK 23.2 series. For details, see MOSK documentation: Upgrade OpenStack.

4(1,2,3,4,5,6,7)

Only Cinder API V3 is supported.

5(1,2,3,4,5)

Since Container Cloud 2.27.3 (Cluster release 16.2.3), the VMware vSphere configuration is unsupported. For details, see Deprecation notes.
VMware vSphere is supported on RHEL 8.7 or Ubuntu 20.04.
RHEL 8.7 is generally available since Cluster releases 16.0.0 and 14.1.0. Before these Cluster releases, it is supported within the Technology Preview features scope.
For Ubuntu deployments, Packer builds a vSphere virtual machine template that is based on Ubuntu 20.04 with kernel 5.15.0-116-generic. If you build a VM template manually, we recommend installing the same kernel version 5.15.0-116-generic.

6(1,2,3,4)

Ceph Pacific supported in 2.23.0 is automatically updated to Quincy during cluster update to 2.24.0.
Ceph Pacific 16.2.11 and Rook 1.0.0-20230120144247 apply to major Cluster releases 12.7.0 and 11.7.0 only.

7(1,2,3)

Attachment of non Container Cloud based MKE clusters is supported only for vSphere-based management clusters on Ubuntu 20.04. Since Container Cloud 2.27.3 (Cluster release 16.2.3), the vSphere-based configuration is unsupported. For details, see Deprecation notes.

8(1,2,3,4,5)

The kernel version of the host operating system is validated by Mirantis and confirmed to be working for the supported use cases. Usage of custom kernel versions or third-party vendor-provided kernels, such as FIPS-enabled, assume full responsibility for validating the compatibility of components in such environments.

9(1,2,3,4,5,6,7,8,9,10,11)

On non-MOSK clusters, Ubuntu 22.04 is installed by default on management and managed clusters. Ubuntu 20.04 is not supported.
On MOSK clusters:
- Since Container Cloud 2.28.0 (Cluster releases 17.3.0), Ubuntu 22.04 is generally available for managed clusters. All existing deployments based on Ubuntu 20.04 must be upgraded to 22.04 within the course of 2.28.x. Otherwise, update of managed clusters to 2.29.0 will become impossible and management cluster update to 2.29.1 will be blocked.
- Before Container Cloud 2.28.0 (Cluster releases 17.2.0, 16.2.0, or earlier), Ubuntu 22.04 is installed by default on management clusters only. And Ubuntu 20.04 is the only supported distribution for managed clusters.

10(1,2,3,4,5,6,7,8)

In Container Cloud 2.27.1, docker-ee-cli is updated to 23.0.13 for MCR 23.0.11 and 23.0.9 to fix several CVEs.

11(1,2,3,4)

In Container Cloud 2.29.1 (Cluster releases 17.3.6, 16.4.1, and 16.3.6), docker-ee-cli is updated to 23.0.17 on MOSK clusters (MCR 23.0.15) and 25.0.9m1 on management clusters (MCR 25.0.8) to fix several CVEs.

12(1,2,3,4,5,6)

In Container Cloud 2.29.3 (Cluster releases 17.3.8, 16.4.3, and 16.3.8), docker-ee-cli is updated to 23.0.18 on MOSK clusters (MCR 23.0.15) and 25.0.10 on management clusters (MCR 25.0.8) to fix several CVEs.

See also

Release Notes

Container Cloud web UI browser compatibility¶

The Container Cloud web UI runs in the browser, separate from any backend software. As such, Mirantis aims to support browsers separately from the backend software in use, although each Container Cloud release is tested with specific browser versions.

Mirantis currently supports the following web browsers for the Container Cloud web UI:

Browser	Supported version	Release date	Supported operating system
Firefox	94.0 or newer	November 2, 2021	Windows, macOS
Google Chrome	96.0.4664 or newer	November 15, 2021	Windows, macOS
Microsoft Edge	95.0.1020 or newer	October 21, 2021	Windows

Caution

This table does not apply to third-party web UIs such as the StackLight or Keycloak endpoints that are available through the Container Cloud web UI. Refer to the official documentation of the corresponding third-party component for details about its supported browsers versions.

To ensure the best user experience, Mirantis recommends that you use the latest version of any of the supported browsers. The use of other browsers or older versions of the browsers we support can result in rendering issues, and can even lead to glitches and crashes in the event that the Container Cloud web UI does not support some JavaScript language features or browser web APIs.

Important

Mirantis does not tie browser support to any particular Container Cloud release.

Mirantis strives to leverage the latest in browser technology to build more performant client software, as well as ensuring that our customers benefit from the latest browser security updates. To this end, our strategy is to regularly move our supported browser versions forward, while also lagging behind the latest releases by approximately one year to give our customers a sufficient upgrade buffer.

See also

Release Notes

Release Notes¶

Major and patch versions update path

Releases summary¶
Container Cloud release	Release date	Supported Cluster releases	Summary
2.29.5	Jul 14, 2025	17.3.10 16.4.5 16.3.10 17.4.0 16.4.0 17.3.0 16.3.0 17.3.9 17.3.8 17.3.7 17.3.6 17.3.5 17.3.4 16.4.4 16.4.3 16.4.2 16.4.1 16.3.9 16.3.8 16.3.7 16.3.6 16.3.5 16.3.4	Container Cloud 2.29.5 is the fifth patch release of the 2.29.x release series that contains the following updates: Support for the patch Cluster releases 16.4.5, 16.3.10, and 17.3.10 that represents MOSK patch release 24.3.7. Support for Mirantis Kubernetes Engine to 3.7.23 Support for `docker-ee-cli` 23.0.18 on MOSK clusters (MCR 23.0.15) and 25.0.10 on management clusters (MCR 25.0.8) Bare metal: update of Ubuntu mirror from ubuntu-2025-05-26-003900 to ubuntu-2025-06-23-003900 along with update of minor kernel version from 5.15.0-140-generic to 5.15.0-142-generic Mandatory migration of container runtime from Docker to containerd Security fixes for CVEs in images
2.29.4	Jun 16, 2025	17.3.9 16.4.4 16.3.9 17.4.0 16.4.0 17.3.0 16.3.0 17.3.8 17.3.7 17.3.6 17.3.5 17.3.4 16.4.3 16.4.2 16.4.1 16.3.8 16.3.7 16.3.6 16.3.5 16.3.4	Container Cloud 2.29.4 is the fourth patch release of the 2.29.x release series that contains the following updates: Support for the patch Cluster releases 16.4.4, 16.3.9, and 17.3.9 that represents MOSK patch release 24.3.6. Support for Mirantis Kubernetes Engine to 3.7.23 Support for `docker-ee-cli` 23.0.18 on MOSK clusters (MCR 23.0.15) and 25.0.10 on management clusters (MCR 25.0.8) Bare metal: update of Ubuntu mirror from ubuntu-2025-03-31-003900 to ubuntu-2025-05-26-003900 along with update of minor kernel version from 5.15.0-135-generic to 5.15.0-140-generic Mandatory migration of container runtime from Docker to containerd Security fixes for CVEs in images
2.29.3	May 20, 2025	17.3.8 16.4.3 16.3.8 17.4.0 16.4.0 17.3.0 16.3.0 17.3.7 17.3.6 17.3.5 17.3.4 16.4.2 16.4.1 16.3.7 16.3.6 16.3.5 16.3.4	Container Cloud 2.29.3 is the third patch release of the 2.29.x release series that contains the following updates: Support for the patch Cluster releases 16.4.3, 16.3.8, and 17.3.8 that represents MOSK patch release 24.3.5. Support for Mirantis Kubernetes Engine to 3.7.22 Support for `docker-ee-cli` 23.0.18 on MOSK clusters (MCR 23.0.15) and 25.0.10 on management clusters (MCR 25.0.8) Mandatory migration of container runtime from Docker to containerd Security fixes for CVEs in images
2.29.2	Apr 22, 2025	17.3.7 16.4.2 16.3.7 17.4.0 16.4.0 17.3.0 16.3.0 17.3.6 17.3.5 17.3.4 16.4.1 16.3.6 16.3.5 16.3.4	Container Cloud 2.29.2 is the second patch release of the 2.29.x release series that contains the following updates: Support for the patch Cluster releases 16.4.2, 16.3.7, and 17.3.7 that represents MOSK patch release 24.3.4. Support for Mirantis Kubernetes Engine to 3.7.20 Support for `docker-ee-cli` 23.0.17 on MOSK clusters (MCR 23.0.15) and 25.0.9m1 on management clusters (MCR 25.0.8) Mandatory migration of container runtime from Docker to containerd Bare metal: update of Ubuntu mirror to ubuntu-2025-03-31-003900 along with update of minor kernel version to 5.15.0-135-generic Ubuntu base image: support for utils that extend NVMe provisioning options Security fixes for CVEs in images
2.29.1	Mar 26, 2025	17.3.6 16.4.1 16.3.6 17.4.0 16.4.0 17.3.0 16.3.0 17.3.5 17.3.4 16.3.5 16.3.4	Container Cloud 2.29.1 is the first patch release of the 2.29.x release series that introduces the following updates: Support for the patch Cluster releases 16.4.1, 16.3.6, and 17.3.6 that represents MOSK patch release 24.3.3. Support for Mirantis Kubernetes Engine to 3.7.20 Support for `docker-ee-cli` 23.0.17 on MOSK clusters (MCR 23.0.15) and 25.0.9m1 on management clusters (MCR 25.0.8) Mandatory migration of container runtime from Docker to containerd Bare metal: update of Ubuntu mirror to ubuntu-2025-03-05-003900 along with update of minor kernel version to 5.15.0-134-generic Security fixes for CVEs in images
2.29.0	Mar 11, 2025	17.4.0 16.4.0 17.3.0 16.3.0 17.3.5 16.3.5 17.3.4 16.3.4	Improvements in the CIS Benchmark compliance for Ubuntu Linux 22.04 LTS v2.0.0 L1 Server Support for MKE 3.7.19 Support for MCR 25.0.8 Switch of the default container runtime from Docker to containerd and mandatory migration to containerd `BareMetalHostInventory` instead of `BareMetalHost` Validation of the `Subnet` object changes against allocated IP addresses Improvements in calculation of update estimates using `ClusterUpdatePlan`
2.28.5	Feb 03, 2025	17.3.5 16.3.5 17.3.0 16.3.0 17.2.0 16.2.0 17.3.4 17.2.7 17.2.6 17.2.5 17.2.4 17.2.3 16.3.4 16.3.3 16.3.2 16.3.1 16.2.7 16.2.6 16.2.5 16.2.4 16.2.3	Container Cloud 2.28.5 is the fifth patch release of the 2.28.x release series that introduces the following updates: Support for the patch Cluster release 16.3.5 and 17.3.5 that represents MOSK patch release 24.3.2. Support for Mirantis Kubernetes Engine to 3.7.18 and Mirantis Container Runtime 23.0.15, which includes containerd 1.6.36. Optional migration of container runtime from Docker to containerd. Bare metal: update of Ubuntu mirror to ubuntu-2025-01-08-003900 along with update of minor kernel version to 5.15.0-130-generic. Security fixes for CVEs in images.
2.28.4	Jan 06, 2025	17.3.4 16.3.4 17.3.0 16.3.0 17.2.0 16.2.0 17.2.7 17.2.6 17.2.5 17.2.4 17.2.3 16.3.3 16.3.2 16.3.1 16.2.7 16.2.6 16.2.5 16.2.4 16.2.3	Container Cloud 2.28.4 is the fourth patch release of the 2.28.x release series that introduces the following updates: Support for the patch Cluster release 16.3.4 and 17.3.4 that represents MOSK patch release 24.3.1. Support for Mirantis Kubernetes Engine to 3.7.17 and Mirantis Container Runtime 23.0.15, which includes containerd 1.6.36. Optional migration of container runtime from Docker to containerd. Bare metal: update of Ubuntu mirror to ubuntu-2024-12-05-003900 along with update of minor kernel version to 5.15.0-126-generic. Security fixes for CVEs in images. OpenStack provider: suspension of support for cluster deployment and update
2.28.3	Dec 09, 2024	17.2.7 16.3.3 16.2.7 17.3.0 16.3.0 17.2.0 16.2.0 17.2.6 17.2.5 17.2.4 17.2.3 16.3.2 16.3.1 16.2.6 16.2.5 16.2.4 16.2.3	Container Cloud 2.28.3 is the third patch release of the 2.28.x release series that introduces the following updates: Support for the patch Cluster release 16.3.3. Support for the patch Cluster releases 16.2.7 and 17.2.7 that represents MOSK patch release 24.2.5. Bare metal: update of Ubuntu mirror to ubuntu-2024-11-18-003900 along with update of minor kernel version to 5.15.0-125-generic. Security fixes for CVEs in images.
2.28.2	Nov 18, 2024	17.2.6 16.3.2 16.2.6 17.3.0 16.3.0 17.2.0 16.2.0 17.2.5 17.2.4 17.2.3 16.3.1 16.2.5 16.2.4 16.2.3	Container Cloud 2.28.2 is the second patch release of the 2.28.x release series that introduces the following updates: Support for the patch Cluster release 16.3.2. Support for the patch Cluster releases 16.2.6 and 17.2.6 that represents MOSK patch release 24.2.4. Support for MKE 3.7.16. Bare metal: update of Ubuntu mirror to ubuntu-2024-10-28-012906 along with update of minor kernel version to 5.15.0-124-generic. Security fixes for CVEs in images.
2.28.1	Oct 30, 2024	17.2.5 16.3.1 16.2.5 17.3.0 16.3.0 17.2.0 16.2.0 17.2.4 17.2.3 16.2.4 16.2.3	Container Cloud 2.28.1 is the first patch release of the 2.28.x release series that introduces the following updates: Support for the patch Cluster release 16.3.1. Support for the patch Cluster releases 16.2.5 and 17.2.5 that represents MOSK patch release 24.2.3. Support for MKE 3.7.15. Bare metal: update of Ubuntu mirror to ubuntu-2024-10-14-013948 along with update of minor kernel version to 5.15.0-122-generic. Security fixes for CVEs in images.
2.28.0	Oct 16, 2024	17.3.0 16.3.0 17.2.0 16.2.0 17.2.4 17.2.3 16.2.4 16.2.3	General availability for Ubuntu 22.04 on MOSK clusters Improvements in the CIS Benchmark compliance for Ubuntu Linux 22.04 LTS v2.0.0 L1 Server Support for MKE 3.7.12 on clusters following the major update path Support for MCR 23.0.14 Update group for controller nodes Reboot of machines using update groups Amendments for the `ClusterUpdatePlan` object Refactoring of delayed auto-update of a management cluster Self-diagnostics for management and managed clusters Configuration of groups in auditd Container Cloud web UI enhancements for the bare metal provider Day-2 operations for bare metal: Updating modules Configuration enhancements for modules StackLight: Monitoring of LCM issues Refactoring of StackLight expiration alerts Documentation enhancements

- Cluster release is deprecated and will become unsupported in one of the following Container Cloud releases.

Container Cloud releases¶

This section outlines the release notes for the Mirantis Container Cloud GA release. Within the scope of the Container Cloud GA release, major releases are being published continuously with new features, improvements, and critical issues resolutions to enhance the Container Cloud GA version. Between major releases, patch releases that incorporate fixes for CVEs of high and critical severity are being delivered. For details, see Container Cloud releases, Cluster releases (managed), and Patch releases.

Once a new Container Cloud release is available, a management cluster automatically upgrades to a newer consecutive release unless this cluster contains managed clusters with a Cluster release unsupported by the newer Container Cloud release. For more details about the Container Cloud release mechanism, see Reference Architecture: Release Controller.

2.29.5 (current)¶

The Container Cloud patch release 2.29.5, which is based on the 2.29.0 major release, provides the following updates:

Support for the patch Cluster releases 16.3.10 and 16.4.5
Support for the patch Cluster release 17.3.10 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 24.3.7
Support for Mirantis Kubernetes Engine to 3.7.23
Support for docker-ee-cli 23.0.18 on MOSK clusters (MCR 23.0.15) and 25.0.10 on management clusters (MCR 25.0.8)
Bare metal: update of Ubuntu mirror from ubuntu-2025-05-26-003900 to ubuntu-2025-06-23-003900 along with update of minor kernel version from 5.15.0-140-generic to 5.15.0-142-generic
Mandatory migration of container runtime from Docker to containerd
Security fixes for CVEs in images

This patch release also supports the latest major Cluster releases 17.4.0 and 16.4.0. And it does not support greenfield deployments based on deprecated Cluster releases. Use the latest available Cluster releases instead.

For main deliverables of the parent Container Cloud release of 2.29.0, refer to 2.29.0.

For the list of MOSK addressed issues, if any, see MOSK documentation: Release notes 24.3.7.

Update notes¶

This section describes the specific actions you as a cloud operator need to complete before or after your Container Cloud cluster update to the Cluster releases 17.3.10, 16.3.10, or 16.4.5.

Consider the information below as a supplement to the generic update procedures published in MOSK Operations Guide: Automatic upgrade of a management cluster and Update to a patch version.

Pre-update actions¶

Ensure that local-volume-provisioner is healthy¶

Due to the known issue 50561, after machine disablement and consequent re-enablement, persistent volumes provisioned by local-volume-provisioner that are not currently in use by any pod may cause the local-volume-provisioner pod on such machine to switch to the CrashLoopBackOff state.

To ensure that local-volume-provisioner does not block cluster update and is healthy, either verify the cluster status in the Container Cloud web UI or run the following command:

kubectl -n kube-system get pods

Example of system response extract with healthy local-volume-provisioner:

local-volume-provisioner-dg98z 1/1 Running 0 66d

Example of system response extract with unhealthy local-volume-provisioner:

local-volume-provisioner-h5lrc   0/1   CrashLoopBackOff   33 (2m3s ago)   90m

If local-volume-provisioner is unhealthy, proceed with the workaround provided in the 50561 issue description.

Post-update actions¶

Mandatory migration of container runtime from Docker to containerd¶

Migration of container runtime from Docker to containerd, which is implemented for existing management and managed clusters, becomes mandatory in the scope of Container Cloud 2.29.x. Otherwise, the management cluster update to Container Cloud 2.30.0 will be blocked.

The use of containerd allows for better Kubernetes performance and component update without pod restart when applying fixes for CVEs. For the migration procedure, refer to MOSK Operations Guide: Migrate container runtime from Docker to containerd.

Important

Container runtime migration involves machine cordoning and draining.

Security notes¶

In total, since Container Cloud 2.29.4, 151 Common Vulnerabilities and Exposures (CVE) have been fixed in 2.29.5: 10 of critical and 141 of high severity.

The table below includes the total numbers of addressed unique and common CVEs in images by product component. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	Critical	High	Total
Kaas core	Unique	2	5	7
Kaas core	Common	8	39	47
StackLight	Unique	2	34	36
StackLight	Common	2	102	104

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 24.3.7: Security notes.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.29.5 including the Cluster releases 17.3.10, 16.3.10, and 16.4.5. For the known issues in the related MOSK release, see MOSK release notes 24.3.7: Known issues.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

The issue occurs when two services share the same external IP address and have the same externalTrafficPolicy value. Initially, the services have the external IP address assigned and are accessible. After modifying the externalTrafficPolicy value for both services from Cluster to Local, the first service that has been changed remains with no external IP address assigned. Though, the second service, which was changed later, has the external IP assigned as expected.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

Ceph¶

[50637] Ceph creates second miracephnodedisable object during node disabling¶

During managed cluster update, if some node is being disabled and at the same time ceph-maintenance-controller is restarted, a second miracephnodedisable object is erroneously created for the node. As a result, the second object fails in the Cleaning state, which blocks managed cluster update.

Workaround

On the affected managed cluster, obtain the list of miracephnodedisable objects:

kubectl get miracephnodedisable -n ceph-lcm-mirantis

The system response must contain one completed and one failed miracephnodedisable object for the node being disabled. For example:

NAME                                               AGE   NODE NAME                                        STATE      LAST CHECK             ISSUE
nodedisable-353ccad2-8f19-4c11-95c9-a783abb531ba   58m   kaas-node-91207a35-3200-41d1-9ba9-388500970981   Ready      2025-03-06T22:04:48Z
nodedisable-58bbf563-1c76-4319-8c28-363d73a5efef   57m   kaas-node-91207a35-3200-41d1-9ba9-388500970981   Cleaning   2025-03-07T11:59:27Z   host clean up Job 'ceph-lcm-mirantis/host-cleanup-nodedisable-58bbf563-1c76-4319-8c28-363d73a5efef' is failed, check logs

Remove the failed miracephnodedisable object. For example:

kubectl delete miracephnodedisable -n ceph-lcm-mirantis nodedisable-58bbf563-1c76-4319-8c28-363d73a5efef

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Update of a managed cluster based on bare metal and Ceph enabled fails with PersistentVolumeClaim getting stuck in the Pending state for the prometheus-server StatefulSet and the MountVolume.MountDevice failed for volume warning in the StackLight event logs.

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

LCM¶

[50561] The local-volume-provisioner pod switches to CrashLoopBackOff¶

After machine disablement and consequent re-enablement, persistent volumes (PVs) provisioned by local-volume-provisioner may cause the local-volume-provisioner pod on such machine to switch to the CrashLoopBackOff state.

This occurs because re-enabling the machine results in a new node UID being assigned in Kubernetes. As a result, the owner ID of the volumes provisioned by local-volume-provisioner no longer matches the new node UID. Although the volumes remain in the correct system paths, local-volume-provisioner detects a mismatch in ownership, leading to an unhealthy service state.

Workaround:

Identify the ID of the affected local-volume-provisioner:

kubectl -n kube-system get pods

Example of system response extract:

local-volume-provisioner-h5lrc   0/1   CrashLoopBackOff   33 (2m3s ago)   90m

In the local-volume-provisioner logs, identify the affected PVs. For example:

kubectl logs -n kube-system local-volume-provisioner-h5lrc | less

Example of system response extract:

E0304 23:21:31.455148    1 discovery.go:221] Failed to discover local volumes:
5 error(s) while discovering volumes: [error creating PV "local-pv-1d04ed53"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol04":
persistentvolumes "local-pv-1d04ed53" already exists error creating PV "local-pv-ce2dfc24"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol01":
persistentvolumes "local-pv-ce2dfc24" already exists error creating PV "local-pv-bcb9e4bd"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol02":
persistentvolumes "local-pv-bcb9e4bd" already exists error creating PV "local-pv-c5924ada"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol03":
persistentvolumes "local-pv-c5924ada" already exists error creating PV "local-pv-7c7150cf"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol00":
persistentvolumes "local-pv-7c7150cf" already exists]

Update the pv.kubernetes.io/provisioned-by annotation for all PVs that are mentioned in the already exists errors on the enabled node. The annotation must have the local-volume-provisioner-<K8S-NODE-NAME>-<K8S-NODE-UID> format.

To obtain the node UID:
```
kubectl get node <K8S-NODE-NAME> -o jsonpath='{.metadata.uid}'
```
To edit annotations on the volumes:
```
kubectl edit pv <PV-NAME>
```
For example:
```
kubectl edit pv local-pv-ce2dfc24
```

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

StackLight¶

[43474] Custom Grafana dashboards are corrupted¶

Custom Grafana panels and dashboards may be corrupted after automatic migration of deprecated Angular-based plugins to the React-based ones. For details, see MOSK Deprecation Notes: Angular plugins in Grafana dashboards and the post-update step Back up custom Grafana dashboards in Container Cloud 2.28.4 update notes.

To work around the issue, manually adjust the affected dashboards to restore their custom appearance.

Container Cloud web UI¶

[50181] Failure to deploy a compact cluster¶

A compact MOSK cluster fails to be deployed through the Container Cloud web UI due to inability to add any label to the control plane machines along with inability to change dedicatedControlPlane: false using the web UI.

To work around the issue, manually add the required labels using CLI. Once done, the cluster deployment resumes.

[50168] Inability to use a new project right after creation¶

A newly created project does not display all available tabs in the Container Cloud web UI and contains different access denied errors during first five minutes after creation.

To work around the issue, refresh the browser in five minutes after the project creation.

Artifacts¶

This section lists the artifacts of components included in the Container Cloud patch release 2.29.5. For artifacts of the Cluster releases introduced in 2.29.5, see patch Cluster releases 17.3.10, 16.3.10, and 16.4.5.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries	ironic-python-agent.initramfs ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-caracal-jammy-debug-20250624163043
	ironic-python-agent.kernel ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-caracal-jammy-debug-20250624163043
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-167-e7a55fd.tgz
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.42.23.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.42.23.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.42.23.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.42.23.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.42.23.tgz
Docker images	ambassador ^Updated	mirantis.azurecr.io/core/external/nginx:1.42.23
	baremetal-dnsmasq	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-29-alpine-20250520130542
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-2-29-alpine-20250617090210
	bm-collective ^Updated	mirantis.azurecr.io/bm/bm-collective:base-2-29-alpine-20250617090858
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.42.23
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:caracal-jammy-20250701070328
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:caracal-jammy-20250701070328
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20240913123302
	kaas-ipam ^Updated	mirantis.azurecr.io/core/kaas-ipam:1.42.23
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-202a68c-20250203183923
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.20-jammy-20241104184039
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20250217103755

Core artifacts¶

Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.42.23.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.42.23.tgz
Helm charts	admission-controller ^Updated	https://binary.mirantis.com/core/helm/admission-controller-1.42.23.tgz
	agent-controller ^Updated	https://binary.mirantis.com/core/helm/agent-controller-1.42.23.tgz
	ceph-kcc-controller ^Updated	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.42.23.tgz
	cert-manager ^Updated	https://binary.mirantis.com/core/helm/cert-manager-1.42.23.tgz
	configuration-collector ^Updated	https://binary.mirantis.com/core/helm/configuration-collector-1.42.23.tgz
	credentials-controller ^Deprecated	https://binary.mirantis.com/core/helm/credentials-controller-1.42.23.tgz
	event-controller ^Updated	https://binary.mirantis.com/core/helm/event-controller-1.42.23.tgz
	host-os-modules-controller ^Updated	https://binary.mirantis.com/core/helm/host-os-modules-controller-1.42.23.tgz
	iam-controller ^Updated	https://binary.mirantis.com/core/helm/iam-controller-1.42.23.tgz
	kaas-exporter ^Updated	https://binary.mirantis.com/core/helm/kaas-exporter-1.42.23.tgz
	kaas-public-api ^Updated	https://binary.mirantis.com/core/helm/kaas-public-api-1.42.23.tgz
	kaas-ui ^Updated	https://binary.mirantis.com/core/helm/kaas-ui-1.42.23.tgz
	lcm-controller ^Updated	https://binary.mirantis.com/core/helm/lcm-controller-1.42.23.tgz
	license-controller ^Updated	https://binary.mirantis.com/core/helm/license-controller-1.42.23.tgz
	machinepool-controller ^Updated	https://binary.mirantis.com/core/helm/machinepool-controller-1.42.23.tgz
	mcc-cache ^Updated	https://binary.mirantis.com/core/helm/mcc-cache-1.42.23.tgz
	mcc-cache-warmup ^Updated	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.42.23.tgz
	openstack-provider ^Deprecated	https://binary.mirantis.com/core/helm/openstack-provider-1.42.23.tgz
	portforward-controller ^Updated	https://binary.mirantis.com/core/helm/portforward-controller-1.42.23.tgz
	rbac-controller ^Updated	https://binary.mirantis.com/core/helm/rbac-controller-1.42.23.tgz
	release-controller ^Updated	https://binary.mirantis.com/core/helm/release-controller-1.42.23.tgz
	scope-controller ^Updated	https://binary.mirantis.com/core/helm/scope-controller-1.42.23.tgz
	secret-controller ^Updated	https://binary.mirantis.com/core/helm/secret-controller-1.42.23.tgz
	user-controller ^Updated	https://binary.mirantis.com/core/helm/user-controller-1.42.23.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.42.23
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.42.23
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.42.23
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-13
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.42.23
	credentials-controller ^Deprecated	mirantis.azurecr.io/core/credentials-controller:1.42.23
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.42.23
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.42.23
	host-os-modules-controller ^Updated	mirantis.azurecr.io/core/host-os-modules-controller:1.42.23
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.42.23
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.42.23
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.42.23
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.42.23
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.42.23
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.42.23
	mcc-cache-warmup ^Updated	mirantis.azurecr.io/core/mcc-cache-warmup:1.42.23
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.42.23
	openstack-cluster-api-controller ^Deprecated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.42.23
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.42.23
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.42.23
	registry ^Updated	mirantis.azurecr.io/lcm/registry:v2.8.1-16
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.42.23
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.42.23
	secret-controller ^Updated	mirantis.azurecr.io/core/secret-controller:1.42.23
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.42.23

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts ^Updated	iam	https://binary.mirantis.com/core/helm/iam-1.42.23.tgz
Docker images	kubectl	mirantis.azurecr.io/general/kubectl:20250307094924
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-d06c869-20250204085201
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.20-jammy-20250218081722
	mcc-keycloak	mirantis.azurecr.io/iam/mcc-keycloak:25.0.6-20250304083438

Unsupported releases¶

Unsupported Container Cloud releases history - 2025¶
Version	Release date	Summary
2.29.4	Jun 16, 2025	Container Cloud 2.29.4 is the fourth patch release of the 2.29.x release series that contains the following updates: Support for the patch Cluster releases 16.4.4, 16.3.9, and 17.3.9 that represents MOSK patch release 24.3.6. Support for Mirantis Kubernetes Engine to 3.7.23 Support for `docker-ee-cli` 23.0.18 on MOSK clusters (MCR 23.0.15) and 25.0.10 on management clusters (MCR 25.0.8) Bare metal: update of Ubuntu mirror from ubuntu-2025-03-31-003900 to ubuntu-2025-05-26-003900 along with update of minor kernel version from 5.15.0-135-generic to 5.15.0-140-generic Mandatory migration of container runtime from Docker to containerd Security fixes for CVEs in images
2.29.3	May 20, 2025	Container Cloud 2.29.3 is the third patch release of the 2.29.x release series that contains the following updates: Support for the patch Cluster releases 16.4.3, 16.3.8, and 17.3.8 that represents MOSK patch release 24.3.5. Support for Mirantis Kubernetes Engine to 3.7.22 Support for `docker-ee-cli` 23.0.18 on MOSK clusters (MCR 23.0.15) and 25.0.10 on management clusters (MCR 25.0.8) Mandatory migration of container runtime from Docker to containerd Security fixes for CVEs in images
2.29.2	Apr 22, 2025	Container Cloud 2.29.2 is the second patch release of the 2.29.x release series that contains the following updates: Support for the patch Cluster releases 16.4.2, 16.3.7, and 17.3.7 that represents MOSK patch release 24.3.4. Support for Mirantis Kubernetes Engine to 3.7.20 Support for `docker-ee-cli` 23.0.17 on MOSK clusters (MCR 23.0.15) and 25.0.9m1 on management clusters (MCR 25.0.8) Mandatory migration of container runtime from Docker to containerd Bare metal: update of Ubuntu mirror to ubuntu-2025-03-31-003900 along with update of minor kernel version to 5.15.0-135-generic Ubuntu base image: support for utils that extend NVMe provisioning options Security fixes for CVEs in images
2.29.1	Mar 26, 2025	Container Cloud 2.29.1 is the first patch release of the 2.29.x release series that introduces the following updates: Support for the patch Cluster releases 16.4.1, 16.3.6, and 17.3.6 that represents MOSK patch release 24.3.3. Support for Mirantis Kubernetes Engine to 3.7.20 Support for `docker-ee-cli` 23.0.17 on MOSK clusters (MCR 23.0.15) and 25.0.9m1 on management clusters (MCR 25.0.8) Mandatory migration of container runtime from Docker to containerd Bare metal: update of Ubuntu mirror to ubuntu-2025-03-05-003900 along with update of minor kernel version to 5.15.0-134-generic Security fixes for CVEs in images
2.29.0	Mar 11, 2025	Improvements in the CIS Benchmark compliance for Ubuntu Linux 22.04 LTS v2.0.0 L1 Server Support for MKE 3.7.19 Support for MCR 25.0.8 Switch of the default container runtime from Docker to containerd `BareMetalHostInventory` instead of `BareMetalHost` Validation of the `Subnet` object changes against allocated IP addresses Improvements in calculation of update estimates using `ClusterUpdatePlan`
2.28.5	Feb 03, 2025	Container Cloud 2.28.5 is the fifth patch release of the 2.28.x release series that introduces the following updates: Support for the patch Cluster release 16.3.5 and 17.3.5 that represents MOSK patch release 24.3.2. Support for Mirantis Kubernetes Engine to 3.7.18 and Mirantis Container Runtime 23.0.15, which includes containerd 1.6.36. Optional migration of container runtime from Docker to containerd. Bare metal: update of Ubuntu mirror to ubuntu-2025-01-08-003900 along with update of minor kernel version to 5.15.0-130-generic. Security fixes for CVEs in images.
2.28.4	Jan 06, 2025	Container Cloud 2.28.4 is the fourth patch release of the 2.28.x release series that introduces the following updates: Support for the patch Cluster release 16.3.4 and 17.3.4 that represents MOSK patch release 24.3.1. Support for Mirantis Kubernetes Engine to 3.7.17 and Mirantis Container Runtime 23.0.15, which includes containerd 1.6.36. Optional migration of container runtime from Docker to containerd. Bare metal: update of Ubuntu mirror to ubuntu-2024-12-05-003900 along with update of minor kernel version to 5.15.0-126-generic. Security fixes for CVEs in images. OpenStack provider: suspension of support for cluster deployment and update

2.29.4¶

The Container Cloud patch release 2.29.4, which is based on the 2.29.0 major release, provides the following updates:

Support for the patch Cluster releases 16.3.9 and 16.4.4
Support for the patch Cluster release 17.3.9 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 24.3.6
Support for Mirantis Kubernetes Engine to 3.7.23
Support for docker-ee-cli 23.0.18 on MOSK clusters (MCR 23.0.15) and 25.0.10 on management clusters (MCR 25.0.8)
Bare metal: update of Ubuntu mirror from ubuntu-2025-03-31-003900 to ubuntu-2025-05-26-003900 along with update of minor kernel version from 5.15.0-135-generic to 5.15.0-140-generic
Mandatory migration of container runtime from Docker to containerd
Security fixes for CVEs in images

For main deliverables of the parent Container Cloud release of 2.29.0, refer to 2.29.0.

For the list of MOSK addressed issues, if any, see MOSK documentation: Release notes 24.3.6.

Update notes¶

This section describes the specific actions you as a cloud operator need to complete before or after your Container Cloud cluster update to the Cluster releases 17.3.9, 16.3.9, or 16.4.4.

Consider the information below as a supplement to the generic update procedures published in MOSK Operations Guide: Automatic upgrade of a management cluster and Update to a patch version.

Pre-update actions¶

Ensure that local-volume-provisioner is healthy¶

To ensure that local-volume-provisioner does not block cluster update and is healthy, either verify the cluster status in the Container Cloud web UI or run the following command:

kubectl -n kube-system get pods

Example of system response extract with healthy local-volume-provisioner:

local-volume-provisioner-dg98z 1/1 Running 0 66d

Example of system response extract with unhealthy local-volume-provisioner:

local-volume-provisioner-h5lrc   0/1   CrashLoopBackOff   33 (2m3s ago)   90m

If local-volume-provisioner is unhealthy, proceed with the workaround provided in the 50561 issue description.

Post-update actions¶

Mandatory migration of container runtime from Docker to containerd¶

Important

Container runtime migration involves machine cordoning and draining.

Security notes¶

In total, since Container Cloud 2.29.3, 46 Common Vulnerabilities and Exposures (CVE) have been fixed in 2.29.4: 2 of critical and 44 of high severity.

The table below includes the total numbers of addressed unique and common CVEs in images by product component. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	Critical	High	Total
Ceph	Unique	1	5	6
Ceph	Common	2	27	29
Kaas core	Unique	0	2	2
Kaas core	Common	0	2	2
StackLight	Unique	0	12	12
StackLight	Common	0	15	15

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 24.3.6: Security notes.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.29.4 including the Cluster releases 17.3.9, 16.3.9, and 16.4.4. For the known issues in the related MOSK release, see MOSK release notes 24.3.6: Known issues.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

Ceph¶

[50637] Ceph creates second miracephnodedisable object during node disabling¶

Workaround

On the affected managed cluster, obtain the list of miracephnodedisable objects:

kubectl get miracephnodedisable -n ceph-lcm-mirantis

The system response must contain one completed and one failed miracephnodedisable object for the node being disabled. For example:

NAME                                               AGE   NODE NAME                                        STATE      LAST CHECK             ISSUE
nodedisable-353ccad2-8f19-4c11-95c9-a783abb531ba   58m   kaas-node-91207a35-3200-41d1-9ba9-388500970981   Ready      2025-03-06T22:04:48Z
nodedisable-58bbf563-1c76-4319-8c28-363d73a5efef   57m   kaas-node-91207a35-3200-41d1-9ba9-388500970981   Cleaning   2025-03-07T11:59:27Z   host clean up Job 'ceph-lcm-mirantis/host-cleanup-nodedisable-58bbf563-1c76-4319-8c28-363d73a5efef' is failed, check logs

Remove the failed miracephnodedisable object. For example:

kubectl delete miracephnodedisable -n ceph-lcm-mirantis nodedisable-58bbf563-1c76-4319-8c28-363d73a5efef

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

LCM¶

[50561] The local-volume-provisioner pod switches to CrashLoopBackOff¶

Workaround:

Identify the ID of the affected local-volume-provisioner:

kubectl -n kube-system get pods

Example of system response extract:

local-volume-provisioner-h5lrc   0/1   CrashLoopBackOff   33 (2m3s ago)   90m

In the local-volume-provisioner logs, identify the affected PVs. For example:

kubectl logs -n kube-system local-volume-provisioner-h5lrc | less

Example of system response extract:

E0304 23:21:31.455148    1 discovery.go:221] Failed to discover local volumes:
5 error(s) while discovering volumes: [error creating PV "local-pv-1d04ed53"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol04":
persistentvolumes "local-pv-1d04ed53" already exists error creating PV "local-pv-ce2dfc24"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol01":
persistentvolumes "local-pv-ce2dfc24" already exists error creating PV "local-pv-bcb9e4bd"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol02":
persistentvolumes "local-pv-bcb9e4bd" already exists error creating PV "local-pv-c5924ada"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol03":
persistentvolumes "local-pv-c5924ada" already exists error creating PV "local-pv-7c7150cf"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol00":
persistentvolumes "local-pv-7c7150cf" already exists]

Update the pv.kubernetes.io/provisioned-by annotation for all PVs that are mentioned in the already exists errors on the enabled node. The annotation must have the local-volume-provisioner-<K8S-NODE-NAME>-<K8S-NODE-UID> format.

To obtain the node UID:
```
kubectl get node <K8S-NODE-NAME> -o jsonpath='{.metadata.uid}'
```
To edit annotations on the volumes:
```
kubectl edit pv <PV-NAME>
```
For example:
```
kubectl edit pv local-pv-ce2dfc24
```

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

StackLight¶

[43474] Custom Grafana dashboards are corrupted¶

To work around the issue, manually adjust the affected dashboards to restore their custom appearance.

Container Cloud web UI¶

[50181] Failure to deploy a compact cluster¶

To work around the issue, manually add the required labels using CLI. Once done, the cluster deployment resumes.

[50168] Inability to use a new project right after creation¶

A newly created project does not display all available tabs in the Container Cloud web UI and contains different access denied errors during first five minutes after creation.

To work around the issue, refresh the browser in five minutes after the project creation.

Artifacts¶

This section lists the artifacts of components included in the Container Cloud patch release 2.29.4. For artifacts of the Cluster releases introduced in 2.29.4, see patch Cluster releases 17.3.9, 16.3.9, and 16.4.4.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries	ironic-python-agent.initramfs ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-caracal-jammy-debug-20250526112742
	ironic-python-agent.kernel ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-caracal-jammy-debug-20250526112742
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-167-e7a55fd.tgz
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.42.19.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.42.19.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.42.19.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.42.19.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.42.19.tgz
Docker images	ambassador ^Updated	mirantis.azurecr.io/core/external/nginx:1.42.19
	baremetal-dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-29-alpine-20250520130542
	baremetal-operator	mirantis.azurecr.io/bm/baremetal-operator:base-2-29-alpine-20250422081418
	bm-collective	mirantis.azurecr.io/bm/bm-collective:base-2-29-alpine-20250303153858
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.42.19
	ironic	mirantis.azurecr.io/openstack/ironic:caracal-jammy-20250217105151
	ironic-inspector	mirantis.azurecr.io/openstack/ironic-inspector:caracal-jammy-20250217105151
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20240913123302
	kaas-ipam ^Updated	mirantis.azurecr.io/core/kaas-ipam:1.42.19
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-202a68c-20250203183923
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.20-jammy-20241104184039
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20250217103755

Core artifacts¶

Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.42.19.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.42.19.tgz
Helm charts	admission-controller ^Updated	https://binary.mirantis.com/core/helm/admission-controller-1.42.19.tgz
	agent-controller ^Updated	https://binary.mirantis.com/core/helm/agent-controller-1.42.19.tgz
	ceph-kcc-controller ^Updated	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.42.19.tgz
	cert-manager ^Updated	https://binary.mirantis.com/core/helm/cert-manager-1.42.19.tgz
	configuration-collector ^Updated	https://binary.mirantis.com/core/helm/configuration-collector-1.42.19.tgz
	credentials-controller ^Deprecated	https://binary.mirantis.com/core/helm/credentials-controller-1.42.19.tgz
	event-controller ^Updated	https://binary.mirantis.com/core/helm/event-controller-1.42.19.tgz
	host-os-modules-controller ^Updated	https://binary.mirantis.com/core/helm/host-os-modules-controller-1.42.19.tgz
	iam-controller ^Updated	https://binary.mirantis.com/core/helm/iam-controller-1.42.19.tgz
	kaas-exporter ^Updated	https://binary.mirantis.com/core/helm/kaas-exporter-1.42.19.tgz
	kaas-public-api ^Updated	https://binary.mirantis.com/core/helm/kaas-public-api-1.42.19.tgz
	kaas-ui ^Updated	https://binary.mirantis.com/core/helm/kaas-ui-1.42.19.tgz
	lcm-controller ^Updated	https://binary.mirantis.com/core/helm/lcm-controller-1.42.19.tgz
	license-controller ^Updated	https://binary.mirantis.com/core/helm/license-controller-1.42.19.tgz
	machinepool-controller ^Updated	https://binary.mirantis.com/core/helm/machinepool-controller-1.42.19.tgz
	mcc-cache ^Updated	https://binary.mirantis.com/core/helm/mcc-cache-1.42.19.tgz
	mcc-cache-warmup ^Updated	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.42.19.tgz
	openstack-provider ^Deprecated	https://binary.mirantis.com/core/helm/openstack-provider-1.42.19.tgz
	portforward-controller ^Updated	https://binary.mirantis.com/core/helm/portforward-controller-1.42.19.tgz
	rbac-controller ^Updated	https://binary.mirantis.com/core/helm/rbac-controller-1.42.19.tgz
	release-controller ^Updated	https://binary.mirantis.com/core/helm/release-controller-1.42.19.tgz
	scope-controller ^Updated	https://binary.mirantis.com/core/helm/scope-controller-1.42.19.tgz
	secret-controller ^Updated	https://binary.mirantis.com/core/helm/secret-controller-1.42.19.tgz
	user-controller ^Updated	https://binary.mirantis.com/core/helm/user-controller-1.42.19.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.42.19
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.42.19
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.42.19
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-13
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.42.19
	credentials-controller ^Deprecated	mirantis.azurecr.io/core/credentials-controller:1.42.19
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.42.19
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.42.19
	host-os-modules-controller ^Updated	mirantis.azurecr.io/core/host-os-modules-controller:1.42.19
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.42.19
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.42.19
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.42.19
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.42.19
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.42.19
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.42.19
	mcc-cache-warmup ^Updated	mirantis.azurecr.io/core/mcc-cache-warmup:1.42.19
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.42.19
	openstack-cluster-api-controller ^Deprecated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.42.19
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.42.19
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.42.19
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-15
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.42.19
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.42.19
	secret-controller ^Updated	mirantis.azurecr.io/core/secret-controller:1.42.19
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.42.19

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts ^Updated	iam	https://binary.mirantis.com/core/helm/iam-1.42.19.tgz
Docker images	kubectl	mirantis.azurecr.io/general/kubectl:20250307094924
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-d06c869-20250204085201
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.20-jammy-20250218081722
	mcc-keycloak	mirantis.azurecr.io/iam/mcc-keycloak:25.0.6-20250304083438

2.29.3¶

The Container Cloud patch release 2.29.3, which is based on the 2.29.0 major release, provides the following updates:

Support for the patch Cluster releases 16.3.8 and 16.4.3
Support for the patch Cluster release 17.3.8 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 24.3.5
Support for Mirantis Kubernetes Engine to 3.7.22
Support for docker-ee-cli 23.0.18 on MOSK clusters (MCR 23.0.15) and 25.0.10 on management clusters (MCR 25.0.8)
Mandatory migration of container runtime from Docker to containerd
Security fixes for CVEs in images

For main deliverables of the parent Container Cloud release of 2.29.0, refer to 2.29.0.

Update notes¶

This section describes the specific actions you as a cloud operator need to complete before or after your Container Cloud cluster update to the Cluster releases 17.3.8, 16.3.8, or 16.4.3.

Consider the information below as a supplement to the generic update procedures published in MOSK Operations Guide: Automatic upgrade of a management cluster and Update to a patch version.

Pre-update actions¶

Ensure that local-volume-provisioner is healthy¶

To ensure that local-volume-provisioner does not block cluster update and is healthy, either verify the cluster status in the Container Cloud web UI or run the following command:

kubectl -n kube-system get pods

Example of system response extract with healthy local-volume-provisioner:

local-volume-provisioner-dg98z 1/1 Running 0 66d

Example of system response extract with unhealthy local-volume-provisioner:

local-volume-provisioner-h5lrc   0/1   CrashLoopBackOff   33 (2m3s ago)   90m

If local-volume-provisioner is unhealthy, proceed with the workaround provided in the 50561 issue description.

Post-update actions¶

Mandatory migration of container runtime from Docker to containerd¶

Important

Container runtime migration involves machine cordoning and draining.

Security notes¶

In total, since Container Cloud 2.29.2, 12 Common Vulnerabilities and Exposures (CVE) have been fixed in 2.29.3: 2 of critical and 10 of high severity.

The table below includes the total numbers of addressed unique and common CVEs in images by product component. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	Critical	High	Total
Ceph	Unique	1	1	2
Ceph	Common	2	1	3
Kaas core	Unique	0	3	3
Kaas core	Common	0	5	5
StackLight	Unique	0	1	1
StackLight	Common	0	4	4

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 24.3.5: Security notes.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.29.3 including the Cluster releases 17.3.8, 16.3.8, and 16.4.3. For the known issues in the related MOSK release, see MOSK release notes 24.3.5: Known issues.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

Ceph¶

[50637] Ceph creates second miracephnodedisable object during node disabling¶

Workaround

On the affected managed cluster, obtain the list of miracephnodedisable objects:

kubectl get miracephnodedisable -n ceph-lcm-mirantis

The system response must contain one completed and one failed miracephnodedisable object for the node being disabled. For example:

NAME                                               AGE   NODE NAME                                        STATE      LAST CHECK             ISSUE
nodedisable-353ccad2-8f19-4c11-95c9-a783abb531ba   58m   kaas-node-91207a35-3200-41d1-9ba9-388500970981   Ready      2025-03-06T22:04:48Z
nodedisable-58bbf563-1c76-4319-8c28-363d73a5efef   57m   kaas-node-91207a35-3200-41d1-9ba9-388500970981   Cleaning   2025-03-07T11:59:27Z   host clean up Job 'ceph-lcm-mirantis/host-cleanup-nodedisable-58bbf563-1c76-4319-8c28-363d73a5efef' is failed, check logs

Remove the failed miracephnodedisable object. For example:

kubectl delete miracephnodedisable -n ceph-lcm-mirantis nodedisable-58bbf563-1c76-4319-8c28-363d73a5efef

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

LCM¶

[50561] The local-volume-provisioner pod switches to CrashLoopBackOff¶

Workaround:

Identify the ID of the affected local-volume-provisioner:

kubectl -n kube-system get pods

Example of system response extract:

local-volume-provisioner-h5lrc   0/1   CrashLoopBackOff   33 (2m3s ago)   90m

In the local-volume-provisioner logs, identify the affected PVs. For example:

kubectl logs -n kube-system local-volume-provisioner-h5lrc | less

Example of system response extract:

E0304 23:21:31.455148    1 discovery.go:221] Failed to discover local volumes:
5 error(s) while discovering volumes: [error creating PV "local-pv-1d04ed53"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol04":
persistentvolumes "local-pv-1d04ed53" already exists error creating PV "local-pv-ce2dfc24"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol01":
persistentvolumes "local-pv-ce2dfc24" already exists error creating PV "local-pv-bcb9e4bd"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol02":
persistentvolumes "local-pv-bcb9e4bd" already exists error creating PV "local-pv-c5924ada"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol03":
persistentvolumes "local-pv-c5924ada" already exists error creating PV "local-pv-7c7150cf"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol00":
persistentvolumes "local-pv-7c7150cf" already exists]

Update the pv.kubernetes.io/provisioned-by annotation for all PVs that are mentioned in the already exists errors on the enabled node. The annotation must have the local-volume-provisioner-<K8S-NODE-NAME>-<K8S-NODE-UID> format.

To obtain the node UID:
```
kubectl get node <K8S-NODE-NAME> -o jsonpath='{.metadata.uid}'
```
To edit annotations on the volumes:
```
kubectl edit pv <PV-NAME>
```
For example:
```
kubectl edit pv local-pv-ce2dfc24
```

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

StackLight¶

[43474] Custom Grafana dashboards are corrupted¶

To work around the issue, manually adjust the affected dashboards to restore their custom appearance.

Container Cloud web UI¶

[50181] Failure to deploy a compact cluster¶

To work around the issue, manually add the required labels using CLI. Once done, the cluster deployment resumes.

[50168] Inability to use a new project right after creation¶

A newly created project does not display all available tabs in the Container Cloud web UI and contains different access denied errors during first five minutes after creation.

To work around the issue, refresh the browser in five minutes after the project creation.

Addressed issues¶

The following issues have been addressed in the Mirantis Container Cloud release 2.29.3 along with the Cluster releases 17.3.8, 16.3.8, and 16.4.3, where applicable. For the list of MOSK addressed issues, if any, see MOSK documentation: Release notes 24.3.5.

[50566] [Ceph] Addressed the issue that caused Ceph upgrade to run very slow during patch or major cluster update due to the upstream Ceph issue 66717.
[51339] [cluster update] Addressed the issue that caused cluster update to be blocked if at least one node of any cluster was not rebooted while applying the update.

Artifacts¶

This section lists the artifacts of components included in the Container Cloud patch release 2.29.3. For artifacts of the Cluster releases introduced in 2.29.3, see patch Cluster releases 17.3.8, 16.3.8, and 16.4.3.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-caracal-jammy-debug-20250331145024
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-caracal-jammy-debug-20250331145024
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-167-e7a55fd.tgz
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.42.18.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.42.18.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.42.18.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.42.18.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.42.18.tgz
Docker images	ambassador ^Updated	mirantis.azurecr.io/core/external/nginx:1.42.18
	baremetal-dnsmasq	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-29-alpine-20250407132604
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-2-29-alpine-20250422081418
	bm-collective	mirantis.azurecr.io/bm/bm-collective:base-2-29-alpine-20250303153858
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.42.18
	ironic	mirantis.azurecr.io/openstack/ironic:caracal-jammy-20250217105151
	ironic-inspector	mirantis.azurecr.io/openstack/ironic-inspector:caracal-jammy-20250217105151
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20240913123302
	kaas-ipam ^Updated	mirantis.azurecr.io/core/kaas-ipam:1.42.18
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-202a68c-20250203183923
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.20-jammy-20241104184039
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20250217103755

Core artifacts¶

Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.42.18.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.42.18.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.42.18.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.42.18.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.42.18.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.42.18.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.42.18.tgz
	credentials-controller ^Deprecated	https://binary.mirantis.com/core/helm/credentials-controller-1.42.18.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.42.18.tgz
	host-os-modules-controller	https://binary.mirantis.com/core/helm/host-os-modules-controller-1.42.18.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.42.18.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.42.18.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.42.18.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.42.18.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.42.18.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.42.18.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.42.18.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.42.18.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.42.18.tgz
	openstack-provider ^Deprecated	https://binary.mirantis.com/core/helm/openstack-provider-1.42.18.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.42.18.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.42.18.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.42.18.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.42.18.tgz
	secret-controller	https://binary.mirantis.com/core/helm/secret-controller-1.42.18.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.42.18.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.42.18
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.42.18
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.42.18
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-13
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.42.18
	credentials-controller ^Deprecated	mirantis.azurecr.io/core/credentials-controller:1.42.18
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.42.18
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.42.18
	host-os-modules-controller ^Updated	mirantis.azurecr.io/core/host-os-modules-controller:1.42.18
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.42.18
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.42.18
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.42.18
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.42.18
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.42.18
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.42.18
	mcc-cache-warmup ^Updated	mirantis.azurecr.io/core/mcc-cache-warmup:1.42.18
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.42.18
	openstack-cluster-api-controller ^Deprecated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.42.18
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.42.18
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.42.18
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-15
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.42.18
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.42.18
	secret-controller ^Updated	mirantis.azurecr.io/core/secret-controller:1.42.18
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.42.18

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts ^Updated	iam	https://binary.mirantis.com/core/helm/iam-1.42.18.tgz
Docker images	kubectl	mirantis.azurecr.io/general/kubectl:20250307094924
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-d06c869-20250204085201
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.20-jammy-20250218081722
	mcc-keycloak	mirantis.azurecr.io/iam/mcc-keycloak:25.0.6-20250304083438

2.29.2¶

The Container Cloud patch release 2.29.2, which is based on the 2.29.0 major release, provides the following updates:

Support for the patch Cluster releases 16.3.7 and 16.4.2
Support for the patch Cluster release 17.3.7 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 24.3.4
Support for Mirantis Kubernetes Engine to 3.7.20
Support for docker-ee-cli 23.0.17 on MOSK clusters (MCR 23.0.15) and 25.0.9m1 on management clusters (MCR 25.0.8)
Mandatory migration of container runtime from Docker to containerd
Bare metal: update of Ubuntu mirror from ubuntu-2025-03-05-003900 to ubuntu-2025-03-31-003900 along with update of minor kernel version from 5.15.0-134-generic to 5.15.0-135-generic
Ubuntu base image: support for utils that extend NVMe provisioning options
Security fixes for CVEs in images

For main deliverables of the parent Container Cloud release of 2.29.0, refer to 2.29.0.

Update notes¶

This section describes the specific actions you as a cloud operator need to complete before or after your Container Cloud cluster update to the Cluster releases 17.3.7, 16.3.7, or 16.4.2.

Consider the information below as a supplement to the generic update procedures published in MOSK Operations Guide: Automatic upgrade of a management cluster and Update to a patch version.

Pre-update actions¶

Ensure that local-volume-provisioner is healthy¶

To ensure that local-volume-provisioner does not block cluster update and is healthy, either verify the cluster status in the Container Cloud web UI or run the following command:

kubectl -n kube-system get pods

Example of system response extract with healthy local-volume-provisioner:

local-volume-provisioner-dg98z 1/1 Running 0 66d

Example of system response extract with unhealthy local-volume-provisioner:

local-volume-provisioner-h5lrc   0/1   CrashLoopBackOff   33 (2m3s ago)   90m

If local-volume-provisioner is unhealthy, proceed with the workaround provided in the 50561 issue description.

Post-update actions¶

Mandatory migration of container runtime from Docker to containerd¶

Important

Container runtime migration involves machine cordoning and draining.

Security notes¶

In total, since Container Cloud 2.29.1, 103 Common Vulnerabilities and Exposures (CVE) have been fixed in 2.29.2: 5 of critical and 98 of high severity.

The table below includes the total numbers of addressed unique and common CVEs in images by product component. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	Critical	High	Total
Ceph	Unique	0	3	3
Ceph	Common	0	13	13
KaaS core	Unique	2	14	16
KaaS core	Common	2	66	68
StackLight	Unique	2	6	8
StackLight	Common	3	19	22

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 24.3.4: Security notes.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.29.2 including the Cluster releases 17.3.7, 16.3.7, and 16.4.2. For the known issues in the related MOSK release, see MOSK release notes 24.3.4: Known issues.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

Ceph¶

[50637] Ceph creates second miracephnodedisable object during node disabling¶

Workaround

On the affected managed cluster, obtain the list of miracephnodedisable objects:

kubectl get miracephnodedisable -n ceph-lcm-mirantis

The system response must contain one completed and one failed miracephnodedisable object for the node being disabled. For example:

NAME                                               AGE   NODE NAME                                        STATE      LAST CHECK             ISSUE
nodedisable-353ccad2-8f19-4c11-95c9-a783abb531ba   58m   kaas-node-91207a35-3200-41d1-9ba9-388500970981   Ready      2025-03-06T22:04:48Z
nodedisable-58bbf563-1c76-4319-8c28-363d73a5efef   57m   kaas-node-91207a35-3200-41d1-9ba9-388500970981   Cleaning   2025-03-07T11:59:27Z   host clean up Job 'ceph-lcm-mirantis/host-cleanup-nodedisable-58bbf563-1c76-4319-8c28-363d73a5efef' is failed, check logs

Remove the failed miracephnodedisable object. For example:

kubectl delete miracephnodedisable -n ceph-lcm-mirantis nodedisable-58bbf563-1c76-4319-8c28-363d73a5efef

[50566] Ceph upgrade is very slow during patch or major cluster update¶

Fixed in 2.29.3 (17.3.8, 16.3.8, and 16.4.3)

Due to the upstream Ceph issue 66717, during CVE upgrade of the Ceph daemon image of Ceph Reef 18.2.4, OSDs may start slow and even fail the starting probe with the following describe output in the rook-ceph-osd-X pod:

 Warning  Unhealthy  57s (x16 over 3m27s)  kubelet  Startup probe failed:
 ceph daemon health check failed with the following output:
> no valid command found; 10 closest matches:
> 0
> 1
> 2
> abort
> assert
> bluefs debug_inject_read_zeros
> bluefs files list
> bluefs stats
> bluestore bluefs device info [<alloc_size:int>]
> config diff
> admin_socket: invalid command

Workaround:

Complete the following steps during every patch or major cluster update of the Cluster releases 17.2.x, 17.3.x, and 17.4.x (until Ceph 18.2.5 becomes supported):

Plan extra time in the maintenance window for the patch cluster update.

Slow starts will still impact the update procedure, but after completing the following step, the recovery process noticeably shortens without affecting the overall cluster state and data responsiveness.
Select one of the following options:
- Before the cluster update, set the noout flag:
```
ceph osd set noout
```
  Once the Ceph OSDs image upgrade is done, unset the flag:
```
ceph osd unset noout
```
- Monitor the Ceph OSDs image upgrade. If the symptoms of slow start appear, set the noout flag as soon as possible. Once the Ceph OSDs image upgrade is done, unset the flag.

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

LCM¶

[50561] The local-volume-provisioner pod switches to CrashLoopBackOff¶

Workaround:

Identify the ID of the affected local-volume-provisioner:

kubectl -n kube-system get pods

Example of system response extract:

local-volume-provisioner-h5lrc   0/1   CrashLoopBackOff   33 (2m3s ago)   90m

In the local-volume-provisioner logs, identify the affected PVs. For example:

kubectl logs -n kube-system local-volume-provisioner-h5lrc | less

Example of system response extract:

E0304 23:21:31.455148    1 discovery.go:221] Failed to discover local volumes:
5 error(s) while discovering volumes: [error creating PV "local-pv-1d04ed53"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol04":
persistentvolumes "local-pv-1d04ed53" already exists error creating PV "local-pv-ce2dfc24"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol01":
persistentvolumes "local-pv-ce2dfc24" already exists error creating PV "local-pv-bcb9e4bd"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol02":
persistentvolumes "local-pv-bcb9e4bd" already exists error creating PV "local-pv-c5924ada"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol03":
persistentvolumes "local-pv-c5924ada" already exists error creating PV "local-pv-7c7150cf"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol00":
persistentvolumes "local-pv-7c7150cf" already exists]

Update the pv.kubernetes.io/provisioned-by annotation for all PVs that are mentioned in the already exists errors on the enabled node. The annotation must have the local-volume-provisioner-<K8S-NODE-NAME>-<K8S-NODE-UID> format.

To obtain the node UID:
```
kubectl get node <K8S-NODE-NAME> -o jsonpath='{.metadata.uid}'
```
To edit annotations on the volumes:
```
kubectl edit pv <PV-NAME>
```
For example:
```
kubectl edit pv local-pv-ce2dfc24
```

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

StackLight¶

[43474] Custom Grafana dashboards are corrupted¶

To work around the issue, manually adjust the affected dashboards to restore their custom appearance.

Cluster update¶

[51339] Cluster upgrade to 2.29.2 is blocked if nodes are not rebooted¶

Fixed in 2.29.3 (17.3.8, 16.3.8, and 16.4.3)

Cluster upgrade from Container Cloud 2.29.1 to 2.29.2 is blocked if at least one node of any cluster is not rebooted while applying upgrade. As a workaround, reboot all cluster nodes.

Container Cloud web UI¶

[50181] Failure to deploy a compact cluster¶

To work around the issue, manually add the required labels using CLI. Once done, the cluster deployment resumes.

[50168] Inability to use a new project right after creation¶

A newly created project does not display all available tabs in the Container Cloud web UI and contains different access denied errors during first five minutes after creation.

To work around the issue, refresh the browser in five minutes after the project creation.

Addressed issues¶

The following issues have been addressed in the Mirantis Container Cloud release 2.29.2 along with the Cluster releases 17.3.7, 16.3.7, and 16.4.2, where applicable. For the list of MOSK addressed issues, if any, see MOSK documentation: Release notes 24.3.4.

[51145] [StackLight] Addressed the issue that caused the PrometheusTargetScrapesDuplicate alert to permanently fire on a management cluster that has sf-notifier enabled.
[50636] [LCM] Addressed the issue that caused the nfs-common package to be deleted during MOSK cluster update. This package is no longer automatically removed from cluster nodes if a MOSK cluster is deployed with the MariaDB backup hosted on an external NFS backend.

Artifacts¶

This section lists the artifacts of components included in the Container Cloud patch release 2.29.2. For artifacts of the Cluster releases introduced in 2.29.2, see patch Cluster releases 17.3.7, 16.3.7, and 16.4.2.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries	ironic-python-agent.initramfs ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-caracal-jammy-debug-20250331145024
	ironic-python-agent.kernel ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-caracal-jammy-debug-20250331145024
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-167-e7a55fd.tgz
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.42.16.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.42.16.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.42.16.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.42.16.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.42.16.tgz
Docker images	ambassador ^Updated	mirantis.azurecr.io/core/external/nginx:1.42.16
	baremetal-dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-29-alpine-20250407132604
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-2-29-alpine-20250407132309
	bm-collective	mirantis.azurecr.io/bm/bm-collective:base-2-29-alpine-20250303153858
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.42.16
	ironic	mirantis.azurecr.io/openstack/ironic:caracal-jammy-20250217105151
	ironic-inspector	mirantis.azurecr.io/openstack/ironic-inspector:caracal-jammy-20250217105151
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20240913123302
	kaas-ipam ^Updated	mirantis.azurecr.io/core/kaas-ipam:1.42.16
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-202a68c-20250203183923
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.20-jammy-20241104184039
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20250217103755

Core artifacts¶

Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.42.16.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.42.16.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.42.16.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.42.16.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.42.16.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.42.16.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.42.16.tgz
	credentials-controller ^Deprecated	https://binary.mirantis.com/core/helm/credentials-controller-1.42.16.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.42.16.tgz
	host-os-modules-controller	https://binary.mirantis.com/core/helm/host-os-modules-controller-1.42.16.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.42.16.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.42.16.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.42.16.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.42.16.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.42.16.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.42.16.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.42.16.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.42.16.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.42.16.tgz
	openstack-provider ^Deprecated	https://binary.mirantis.com/core/helm/openstack-provider-1.42.16.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.42.16.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.42.16.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.42.16.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.42.16.tgz
	secret-controller	https://binary.mirantis.com/core/helm/secret-controller-1.42.16.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.42.16.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.42.16
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.42.16
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.42.16
	cert-manager-controller ^Updated	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-13
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.42.16
	credentials-controller ^Deprecated	mirantis.azurecr.io/core/credentials-controller:1.42.16
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.42.16
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.42.16
	host-os-modules-controller ^Updated	mirantis.azurecr.io/core/host-os-modules-controller:1.42.16
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.42.16
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.42.16
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.42.16
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.42.16
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.42.16
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.42.16
	mcc-cache-warmup ^Updated	mirantis.azurecr.io/core/mcc-cache-warmup:1.42.16
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.42.16
	openstack-cluster-api-controller ^Deprecated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.42.16
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.42.16
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.42.16
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-15
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.42.16
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.42.16
	secret-controller ^Updated	mirantis.azurecr.io/core/secret-controller:1.42.16
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.42.16

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts ^Updated	iam	https://binary.mirantis.com/core/helm/iam-1.42.16.tgz
Docker images	kubectl	mirantis.azurecr.io/general/kubectl:20250307094924
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-d06c869-20250204085201
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.20-jammy-20250218081722
	mcc-keycloak	mirantis.azurecr.io/iam/mcc-keycloak:25.0.6-20250304083438

2.29.1¶

The Container Cloud patch release 2.29.1, which is based on the 2.29.0 major release, provides the following updates:

Support for the patch Cluster releases 16.3.6 and 16.4.1
Support for the patch Cluster release 17.3.6 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 24.3.3
Support for Mirantis Kubernetes Engine to 3.7.20
Support for docker-ee-cli 23.0.17 on MOSK clusters (MCR 23.0.15) and 25.0.9m1 on management clusters (MCR 25.0.8)
Mandatory migration of container runtime from Docker to containerd
Bare metal: update of Ubuntu mirror to ubuntu-2025-03-05-003900 along with update of minor kernel version to 5.15.0-134-generic
Security fixes for CVEs in images

For main deliverables of the parent Container Cloud release of 2.29.0, refer to 2.29.0.

Update notes¶

This section describes the specific actions you as a cloud operator need to complete before or after your Container Cloud cluster update to the Cluster releases 17.3.6, 16.3.6, or 16.4.1.

Consider the information below as a supplement to the generic update procedures published in MOSK Operations Guide: Automatic upgrade of a management cluster and Update to a patch version.

Pre-update actions¶

Update managed clusters to Ubuntu 22.04¶

Management cluster update to Container Cloud 2.29.1 will be blocked if at least one node of any related managed cluster is running Ubuntu 20.04, which reaches end-of-life in April 2025. Moreover, in Container Cloud 2.29.0, the Cluster release update of the Ubuntu 20.04-based managed clusters became impossible, and Ubuntu 22.04 became the only supported version of the operating system.

Therefore, ensure that every node of all your managed clusters are running Ubuntu 22.04 to unblock management cluster update in Container Cloud 2.29.1 and managed cluster update in Container Cloud 2.29.0.

For the update procedure, refer to Mirantis OpenStack for Kubernetes documentation: Bare metal operations - Upgrade an operating system distribution.

Note

Existing management clusters were automatically updated to Ubuntu 22.04 during cluster upgrade to the Cluster release 16.2.0 in Container Cloud 2.27.0. Greenfield deployments of management clusters are also based on Ubuntu 22.04.

Ensure that local-volume-provisioner is healthy¶

To ensure that local-volume-provisioner does not block cluster update and is healthy, either verify the cluster status in the Container Cloud web UI or run the following command:

kubectl -n kube-system get pods

Example of system response extract with healthy local-volume-provisioner:

local-volume-provisioner-dg98z 1/1 Running 0 66d

Example of system response extract with unhealthy local-volume-provisioner:

local-volume-provisioner-h5lrc   0/1   CrashLoopBackOff   33 (2m3s ago)   90m

If local-volume-provisioner is unhealthy, proceed with the workaround provided in the 50561 issue description.

Post-update actions¶

Migration of container runtime from Docker to containerd¶

Since Container Cloud 2.28.4, Mirantis introduced an optional migration of container runtime from Docker to containerd, which is implemented for existing management and managed bare metal clusters. This migration becomes mandatory in the scope of Container Cloud 2.29.x. Otherwise, the management cluster update to Container Cloud 2.30.0 will be blocked.

Important

Container runtime migration involves machine cordoning and draining.

Security notes¶

In total, since Container Cloud 2.29.0, 203 Common Vulnerabilities and Exposures (CVE) have been fixed in 2.29.1: 21 of critical and 182 of high severity.

The table below includes the total numbers of addressed unique and common CVEs in images by product component since Container Cloud 2.29.0. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	Critical	High	Total
Ceph	Unique	1	10	11
Ceph	Common	2	58	60
Kaas core	Unique	7	27	34
Kaas core	Common	13	78	91
StackLight	Unique	3	13	16
StackLight	Common	6	46	52

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 24.3.3: Security notes.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.29.1 including the Cluster releases 17.3.6, 16.3.6, and 16.4.1. For the known issues in the related MOSK release, see MOSK release notes 24.3.3: Known issues.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

Ceph¶

[50637] Ceph creates second miracephnodedisable object during node disabling¶

Workaround

On the affected managed cluster, obtain the list of miracephnodedisable objects:

kubectl get miracephnodedisable -n ceph-lcm-mirantis

The system response must contain one completed and one failed miracephnodedisable object for the node being disabled. For example:

NAME                                               AGE   NODE NAME                                        STATE      LAST CHECK             ISSUE
nodedisable-353ccad2-8f19-4c11-95c9-a783abb531ba   58m   kaas-node-91207a35-3200-41d1-9ba9-388500970981   Ready      2025-03-06T22:04:48Z
nodedisable-58bbf563-1c76-4319-8c28-363d73a5efef   57m   kaas-node-91207a35-3200-41d1-9ba9-388500970981   Cleaning   2025-03-07T11:59:27Z   host clean up Job 'ceph-lcm-mirantis/host-cleanup-nodedisable-58bbf563-1c76-4319-8c28-363d73a5efef' is failed, check logs

Remove the failed miracephnodedisable object. For example:

kubectl delete miracephnodedisable -n ceph-lcm-mirantis nodedisable-58bbf563-1c76-4319-8c28-363d73a5efef

[50566] Ceph upgrade is very slow during patch or major cluster update¶

Fixed in 2.29.3 (17.3.8, 16.3.8, and 16.4.3)

 Warning  Unhealthy  57s (x16 over 3m27s)  kubelet  Startup probe failed:
 ceph daemon health check failed with the following output:
> no valid command found; 10 closest matches:
> 0
> 1
> 2
> abort
> assert
> bluefs debug_inject_read_zeros
> bluefs files list
> bluefs stats
> bluestore bluefs device info [<alloc_size:int>]
> config diff
> admin_socket: invalid command

Workaround:

Complete the following steps during every patch or major cluster update of the Cluster releases 17.2.x, 17.3.x, and 17.4.x (until Ceph 18.2.5 becomes supported):

Plan extra time in the maintenance window for the patch cluster update.

Slow starts will still impact the update procedure, but after completing the following step, the recovery process noticeably shortens without affecting the overall cluster state and data responsiveness.
Select one of the following options:
- Before the cluster update, set the noout flag:
```
ceph osd set noout
```
  Once the Ceph OSDs image upgrade is done, unset the flag:
```
ceph osd unset noout
```
- Monitor the Ceph OSDs image upgrade. If the symptoms of slow start appear, set the noout flag as soon as possible. Once the Ceph OSDs image upgrade is done, unset the flag.

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

LCM¶

[50561] The local-volume-provisioner pod switches to CrashLoopBackOff¶

Workaround:

Identify the ID of the affected local-volume-provisioner:

kubectl -n kube-system get pods

Example of system response extract:

local-volume-provisioner-h5lrc   0/1   CrashLoopBackOff   33 (2m3s ago)   90m

In the local-volume-provisioner logs, identify the affected PVs. For example:

kubectl logs -n kube-system local-volume-provisioner-h5lrc | less

Example of system response extract:

E0304 23:21:31.455148    1 discovery.go:221] Failed to discover local volumes:
5 error(s) while discovering volumes: [error creating PV "local-pv-1d04ed53"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol04":
persistentvolumes "local-pv-1d04ed53" already exists error creating PV "local-pv-ce2dfc24"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol01":
persistentvolumes "local-pv-ce2dfc24" already exists error creating PV "local-pv-bcb9e4bd"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol02":
persistentvolumes "local-pv-bcb9e4bd" already exists error creating PV "local-pv-c5924ada"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol03":
persistentvolumes "local-pv-c5924ada" already exists error creating PV "local-pv-7c7150cf"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol00":
persistentvolumes "local-pv-7c7150cf" already exists]

Update the pv.kubernetes.io/provisioned-by annotation for all PVs that are mentioned in the already exists errors on the enabled node. The annotation must have the local-volume-provisioner-<K8S-NODE-NAME>-<K8S-NODE-UID> format.

To obtain the node UID:
```
kubectl get node <K8S-NODE-NAME> -o jsonpath='{.metadata.uid}'
```
To edit annotations on the volumes:
```
kubectl edit pv <PV-NAME>
```
For example:
```
kubectl edit pv local-pv-ce2dfc24
```

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

StackLight¶

[51145] PrometheusTargetScrapesDuplicate firing on a management cluster¶

Fixed in 2.29.2 (17.3.7, 16.3.7, and 16.4.2)

On management clusters with sf-notifier enabled, the PrometheusTargetScrapesDuplicate alert is permanently firing while sf-notifier runs with no errors.

You can safely disregard the issue because it does not affect cluster health.

[43474] Custom Grafana dashboards are corrupted¶

To work around the issue, manually adjust the affected dashboards to restore their custom appearance.

Container Cloud web UI¶

[50181] Failure to deploy a compact cluster¶

To work around the issue, manually add the required labels using CLI. Once done, the cluster deployment resumes.

[50168] Inability to use a new project right after creation¶

A newly created project does not display all available tabs in the Container Cloud web UI and contains different access denied errors during first five minutes after creation.

To work around the issue, refresh the browser in five minutes after the project creation.

Addressed issues¶

The following issues have been addressed in the Mirantis Container Cloud release 2.29.1 along with the Cluster releases 17.3.6, 16.3.6, and 16.4.1, where applicable. For the list of MOSK addressed issues, if any, see MOSK documentation: Release notes 24.3.3.

[50768] [LCM] Addressed the issue that prevented successful editing of the MCCUpgrade object, which contained the Internal error failed to call webhook: the server could not find the requested resource when trying to save changes in the object.
[50622] [core] Addressed the issue that prevented any user except m:kaas@management-admin to access or modify BareMetalHostInventory objects.
[50287] [bare metal] Addressed the issue that prevented a BareMetalHost object with a Redfish Baseboard Management Controller address to pass the registering phase.
[50140] [Container Cloud web UI] Addressed the issue that prevented the Clusters page for the bare metal provider to display information about the Ceph cluster in the Ceph Clusters tab.

Artifacts¶

This section lists the artifacts of components included in the Container Cloud patch release 2.29.1. For artifacts of the Cluster releases introduced in 2.29.1, see patch Cluster releases 17.3.6, 16.3.6, and 16.4.1.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries	ironic-python-agent.initramfs ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-caracal-jammy-debug-20250305171003
	ironic-python-agent.kernel ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-caracal-jammy-debug-20250305171003
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-167-e7a55fd.tgz
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.42.13.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.42.13.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.42.13.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.42.13.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.42.13.tgz
Docker images	ambassador ^Updated	mirantis.azurecr.io/core/external/nginx:1.42.13
	baremetal-dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-29-alpine-20250319110626
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-2-29-alpine-20250318131222
	bm-collective ^Updated	mirantis.azurecr.io/bm/bm-collective:base-2-29-alpine-20250303153858
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.42.13
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:caracal-jammy-20250217105151
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:caracal-jammy-20250217105151
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20240913123302
	kaas-ipam ^Updated	mirantis.azurecr.io/core/kaas-ipam:1.42.13
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-202a68c-20250203183923
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.20-jammy-20241104184039
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20250217103755

Core artifacts¶

Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.42.13.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.42.13.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.42.13.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.42.13.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.42.13.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.42.13.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.42.13.tgz
	credentials-controller ^Deprecated	https://binary.mirantis.com/core/helm/credentials-controller-1.42.13.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.42.13.tgz
	host-os-modules-controller	https://binary.mirantis.com/core/helm/host-os-modules-controller-1.42.13.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.42.13.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.42.13.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.42.13.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.42.13.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.42.13.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.42.13.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.42.13.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.42.13.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.42.13.tgz
	openstack-provider ^Deprecated	https://binary.mirantis.com/core/helm/openstack-provider-1.42.13.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.42.13.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.42.13.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.42.13.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.42.13.tgz
	secret-controller	https://binary.mirantis.com/core/helm/secret-controller-1.42.13.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.42.13.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.42.13
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.42.13
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.42.13
	cert-manager-controller ^Updated	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-12
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.42.13
	credentials-controller ^Deprecated	mirantis.azurecr.io/core/credentials-controller:1.42.13
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.42.13
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.42.13
	host-os-modules-controller ^Updated	mirantis.azurecr.io/core/host-os-modules-controller:1.42.13
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.42.13
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.42.13
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.42.13
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.42.13
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.42.13
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.42.13
	mcc-cache-warmup	mirantis.azurecr.io/core/mcc-cache-warmup:1.42.13
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.42.13
	openstack-cluster-api-controller ^Deprecated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.42.13
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.42.13
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.42.13
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-15
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.42.13
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.42.13
	secret-controller ^Updated	mirantis.azurecr.io/core/secret-controller:1.42.13
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.42.13

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts ^Updated	iam	https://binary.mirantis.com/core/helm/iam-1.42.13.tgz
Docker images ^Updated	kubectl	mirantis.azurecr.io/general/kubectl:20250307094924
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-d06c869-20250204085201
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.20-jammy-20250218081722
	mcc-keycloak	mirantis.azurecr.io/iam/mcc-keycloak:25.0.6-20250304083438

2.29.0¶

The Mirantis Container Cloud major release 2.29.0:

Introduces support for the Cluster release 17.4.0 that is based on the Cluster release 16.4.0 and represents Mirantis OpenStack for Kubernetes (MOSK) 25.1.
Introduces support for the Cluster release 16.4.0 that is based on Mirantis Container Runtime (MCR) 25.0.8 and Mirantis Kubernetes Engine (MKE) 3.7.19 with Kubernetes 1.27.
Does not support greenfield deployments on deprecated Cluster releases of the 17.3.x and 16.3.x series. Use the latest available Cluster releases of the series instead.

Caution

Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.29.0.

Enhancements¶

This section outlines new features and enhancements introduced in the Container Cloud release 2.29.0.

For the list of enhancements delivered with the Cluster releases introduced by Container Cloud 2.29.0, see 17.4.0 and 16.4.0.
For the list of enhancements delivered with MOSK 25.1 introduced together with Container Cloud 2.29.0, see MOSK release notes 25.1: New features.

BareMetalHostInventory instead of BareMetalHost¶

To allow the operator use the gitops approach, implemented the BareMetalHostInventory resource that must be used instead of BareMetalHost for adding and modifying configuration of bare metal servers.

The BareMetalHostInventory resource monitors and manages the state of a bare metal server and is created for each Machine with all information about machine hardware configuration.

Each BareMetalHostInventory object is synchronized with an automatically created BareMetalHost object, which is now used for internal purposes of the Container Cloud private API.

Caution

Any change in the BareMetalHost object will be overwitten by BareMetalHostInventory.

For any existing BareMetalHost object, a BareMetalHostInventory object is created automatically during cluster update.

Caution

While the Cluster release the management cluster is 16.4.0, BareMetalHostInventory operations are allowed to m:kaas@management-admin only. Once the management cluster is updated to the Cluster release 16.4.1 (or later), this limitation will be lifted.

Learn more

API Reference: BareMetalHostInventory resource

Validation of the Subnet object changes against allocated IP addresses¶

Implemented a validation of the Subnet object changes against already allocated IP addresses. This validation is performed by the Admission Controller. The controller now blocks changes in the Subnet object containing allocated IP addresses that are out of the allocatable IP address space, which is formed by a CIDR address and include/exclude address ranges.

Learn more

Improvements in calculation of update estimates using ClusterUpdatePlan¶

Improved calculation of update estimates for a managed cluster that is managed by the ClusterUpdatePlan object. Each step of ClusterUpdatePlan now has more precise estimates that are based on the following calculations:

The amount and type of components updated between releases during patch updates
The amount of nodes with particular roles in the OpenStack cluster
The number of nodes and storage used in the Ceph cluster

Also, the ClusterUpdatePlan object now contains the releaseNotes field that links to MOSK release notes of the target release.

Learn more

Switch of the default container runtime from Docker to containerd¶

Switched the default container runtime from Docker to containerd on greenfield management and managed clusters. The use of containerd allows for better Kubernetes performance and component update without pod restart when applying fixes for CVEs.

On existing clusters, perform the mandatory migration from Docker to containerd in the scope of Container Cloud 2.29.x. Otherwise, the management cluster update to Container Cloud 2.30.0 will be blocked.

Important

Container runtime migration involves machine cordoning and draining.

Learn more

MOSK Operations Guide: Migrate container runtime from Docker to containerd

Addressed issues¶

The following issues have been addressed in the Mirantis Container Cloud release 2.29.0 along with the Cluster releases 17.4.0 and 16.4.0. For the list of MOSK addressed issues, see MOSK release notes 25.1: Addressed issues.

Note

This section provides descriptions of issues addressed since the last Container Cloud patch release 2.28.5.

For details on addressed issues in earlier patch releases since 2.28.0, which are also included into the major release 2.29.0, refer to 2.28.x patch releases.

[47263] [StackLight] Fixed the issue with configuration inconsistencies for requests and limits between the deprecated resourcesPerClusterSize and resources parameters.
[44193] [StackLight] Fixed the issue with OpenSearch reaching the 85% disk usage watermark on High Availability clusters that use Local Volume Provisioner, which caused the OpenSearch cluster state to switch to Warning or Critical.
[46858] [Container Cloud web UI] Fixed the issue that prevented the drop-down menu from displaying the full list of allowed node labels.
[39437] [LCM] Fixed the issue that caused failure to replace a master node and the Kubelet's NodeReady condition is Unknown message in the machine status on the remaining master nodes.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.29.0 including the Cluster releases 17.4.0 and 16.4.0. For the known issues in the related MOSK release, see MOSK release notes 25.1: Known issues.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[50287] BareMetalHost with a Redfish BMC address is stuck on registering phase¶

Fixed in 2.29.1 (17.3.6, 16.3.6, and 16.4.1)

During addition of a bare metal host containing a Redfish Baseboard Management Controller address with the following exemplary configuration may get stuck during the registering phase:

bmc:
  address: redfish://192.168.1.150/redfish/v1/Systems/1

Workaround:

Open the ironic-config configmap for editing:

KUBECONFIG=mgmt_kubeconfig kubectl -n kaas edit cm ironic-config

In the data:ironic.conf section, add the enabled_firmware_interfaces parameter:

data:
  ironic.conf: |

    [DEFAULT]
    ...
    enabled_firmware_interfaces = redfish,no-firmware
    ...

Restart Ironic:

KUBECONFIG=mgmt_kubeconfig kubectl -n kaas rollout restart deployment/ironic

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

Ceph¶

[50637] Ceph creates second miracephnodedisable object during node disabling¶

Workaround

On the affected managed cluster, obtain the list of miracephnodedisable objects:

kubectl get miracephnodedisable -n ceph-lcm-mirantis

The system response must contain one completed and one failed miracephnodedisable object for the node being disabled. For example:

NAME                                               AGE   NODE NAME                                        STATE      LAST CHECK             ISSUE
nodedisable-353ccad2-8f19-4c11-95c9-a783abb531ba   58m   kaas-node-91207a35-3200-41d1-9ba9-388500970981   Ready      2025-03-06T22:04:48Z
nodedisable-58bbf563-1c76-4319-8c28-363d73a5efef   57m   kaas-node-91207a35-3200-41d1-9ba9-388500970981   Cleaning   2025-03-07T11:59:27Z   host clean up Job 'ceph-lcm-mirantis/host-cleanup-nodedisable-58bbf563-1c76-4319-8c28-363d73a5efef' is failed, check logs

Remove the failed miracephnodedisable object. For example:

kubectl delete miracephnodedisable -n ceph-lcm-mirantis nodedisable-58bbf563-1c76-4319-8c28-363d73a5efef

[50566] Ceph upgrade is very slow during patch or major cluster update¶

Fixed in 2.29.3 (17.3.8, 16.3.8, and 16.4.3)

 Warning  Unhealthy  57s (x16 over 3m27s)  kubelet  Startup probe failed:
 ceph daemon health check failed with the following output:
> no valid command found; 10 closest matches:
> 0
> 1
> 2
> abort
> assert
> bluefs debug_inject_read_zeros
> bluefs files list
> bluefs stats
> bluestore bluefs device info [<alloc_size:int>]
> config diff
> admin_socket: invalid command

Workaround:

Complete the following steps during every patch or major cluster update of the Cluster releases 17.2.x, 17.3.x, and 17.4.x (until Ceph 18.2.5 becomes supported):

Plan extra time in the maintenance window for the patch cluster update.

Slow starts will still impact the update procedure, but after completing the following step, the recovery process noticeably shortens without affecting the overall cluster state and data responsiveness.
Select one of the following options:
- Before the cluster update, set the noout flag:
```
ceph osd set noout
```
  Once the Ceph OSDs image upgrade is done, unset the flag:
```
ceph osd unset noout
```
- Monitor the Ceph OSDs image upgrade. If the symptoms of slow start appear, set the noout flag as soon as possible. Once the Ceph OSDs image upgrade is done, unset the flag.

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

LCM¶

[50561] The local-volume-provisioner pod switches to CrashLoopBackOff¶

Workaround:

Identify the ID of the affected local-volume-provisioner:

kubectl -n kube-system get pods

Example of system response extract:

local-volume-provisioner-h5lrc   0/1   CrashLoopBackOff   33 (2m3s ago)   90m

In the local-volume-provisioner logs, identify the affected PVs. For example:

kubectl logs -n kube-system local-volume-provisioner-h5lrc | less

Example of system response extract:

E0304 23:21:31.455148    1 discovery.go:221] Failed to discover local volumes:
5 error(s) while discovering volumes: [error creating PV "local-pv-1d04ed53"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol04":
persistentvolumes "local-pv-1d04ed53" already exists error creating PV "local-pv-ce2dfc24"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol01":
persistentvolumes "local-pv-ce2dfc24" already exists error creating PV "local-pv-bcb9e4bd"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol02":
persistentvolumes "local-pv-bcb9e4bd" already exists error creating PV "local-pv-c5924ada"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol03":
persistentvolumes "local-pv-c5924ada" already exists error creating PV "local-pv-7c7150cf"
for volume at "/mnt/local-volumes/openstack-operator/bind-mounts/vol00":
persistentvolumes "local-pv-7c7150cf" already exists]

Update the pv.kubernetes.io/provisioned-by annotation for all PVs that are mentioned in the already exists errors on the enabled node. The annotation must have the local-volume-provisioner-<K8S-NODE-NAME>-<K8S-NODE-UID> format.

To obtain the node UID:
```
kubectl get node <K8S-NODE-NAME> -o jsonpath='{.metadata.uid}'
```
To edit annotations on the volumes:
```
kubectl edit pv <PV-NAME>
```
For example:
```
kubectl edit pv local-pv-ce2dfc24
```

[50768] Failure to update the MCCUpgrade object¶

Fixed in 2.29.1 (17.3.6, 16.3.6, and 16.4.1)

While editing the MCCUpgrade object, the following error occurs when trying to save changes:

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure",
"message":"Internal error occurred: failed calling webhook \"mccupgrades.kaas.mirantis.com\":
failed to call webhook: the server could not find the requested resource",
"reason":"InternalError",
"details":{"causes":[{"message":"failed calling webhook \"mccupgrades.kaas.mirantis.com\":
failed to call webhook: the server could not find the requested resource"}]},"code":500}

To work around the issue, remove the name: mccupgrades.kaas.mirantis.com entry from mutatingwebhookconfiguration:

kubectl --kubeconfig kubeconfig edit mutatingwebhookconfiguration admission-controller

Example configuration:

- admissionReviewVersions:
  - v1
  - v1beta1
  clientConfig:
    caBundle: <REDACTED>
    service:
      name: admission-controller
      namespace: kaas
      path: /mccupgrades
      port: 443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: mccupgrades.kaas.mirantis.com
  namespaceSelector: {}
  objectSelector: {}
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - kaas.mirantis.com
    apiVersions:
    - v1alpha1
    operations:
    - CREATE
    - UPDATE
    resources:
    - mccupgrades
    scope: '*'
  sideEffects: NoneOnDryRun
  timeoutSeconds: 5

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

StackLight¶

[51145] PrometheusTargetScrapesDuplicate firing on a management cluster¶

Fixed in 2.29.2 (17.3.7, 16.3.7, and 16.4.2)

On management clusters with sf-notifier enabled, the PrometheusTargetScrapesDuplicate alert is permanently firing while sf-notifier runs with no errors.

You can safely disregard the issue because it does not affect cluster health.

[43474] Custom Grafana dashboards are corrupted¶

To work around the issue, manually adjust the affected dashboards to restore their custom appearance.

Container Cloud web UI¶

[50181] Failure to deploy a compact cluster¶

To work around the issue, manually add the required labels using CLI. Once done, the cluster deployment resumes.

[50168] Inability to use a new project right after creation¶

A newly created project does not display all available tabs in the Container Cloud web UI and contains different access denied errors during first five minutes after creation.

To work around the issue, refresh the browser in five minutes after the project creation.

[50140] The Ceph Clusters tab does not display Ceph cluster details¶

Fixed in 2.29.1 (17.3.6, 16.3.6, 16.4.1)

The Clusters page for the bare metal provider does not display information about the Ceph cluster in the Ceph Clusters tab and contains access denied errors.

To work around the issue, verify the Ceph cluster state through CLI. For details, see MOSK documentation: Ceph operations - Verify Ceph.

Components versions¶

The following table lists major components and their versions delivered in Container Cloud 2.29.0. The components that are newly added, updated, deprecated, or removed as compared to 2.28.0, are marked with a corresponding superscript, for example, admission-controller ^Updated.

Component	Application/Service	Version
Bare metal ^Updated	ambassador	1.42.9
	baremetal-dnsmasq	base-2-29-alpine-20250217104113
	baremetal-operator	base-2-29-alpine-20250217104322
	baremetal-provider	1.42.9
	bm-collective	base-2-29-alpine-20250217104943
	cluster-api-provider-baremetal	1.42.9
	ironic	caracal-jammy-20250128120200
	ironic-inspector	caracal-jammy-20250128120200
	ironic-prometheus-exporter	0.1-20240913123302
	kaas-ipam	1.42.9
	kubernetes-entrypoint	1.0.1-202a68c-20250203183923
	mariadb	10.6.20-jammy-20241104184039
	syslog-ng	base-alpine-20250217103755
Container Cloud ^Updated	admission-controller	1.42.9
	agent-controller	1.42.9
	byo-cluster-api-controller	1.42.9
	ceph-kcc-controller	1.42.9
	cert-manager-controller	1.11.0-11
	configuration-collector	1.42.9
	event-controller	1.42.9
	frontend	1.42.9
	golang	1.23.6-alpine3.20
	iam-controller	1.42.9
	kaas-exporter	1.42.9
	kproxy	1.42.9
	lcm-controller	1.42.9
	license-controller	1.42.9
	machinepool-controller	1.42.9
	nginx	1.42.9
	portforward-controller	1.42.9
	rbac-controller	1.42.9
	registry	2.8.1-15
	release-controller	1.42.9
	scope-controller	1.42.9
	secret-controller	1.42.9
	user-controller	1.42.9
IAM ^Updated	iam	1.42.9
	mariadb	10.6.20-jammy-20241104184039
	mcc-keycloak	25.0.6-20241114073807
OpenStack ^Deprecated	host-os-modules-controller	1.42.9
	openstack-cluster-api-controller	1.42.9
	openstack-provider	1.42.9

Artifacts¶

This section lists the artifacts of components included in the Container Cloud release 2.29.0. The components that are newly added, updated, deprecated, or removed as compared to 2.28.0, are marked with a corresponding superscript, for example, admission-controller ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries	ironic-python-agent.initramfs ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-caracal-jammy-debug-20250217102957
	ironic-python-agent.kernel ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-caracal-jammy-debug-20250217102957
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-167-e7a55fd.tgz
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.42.9.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.42.9.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.42.9.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.42.9.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.42.9.tgz
Docker images ^Updated	ambassador	mirantis.azurecr.io/core/external/nginx:1.42.9
	baremetal-dnsmasq	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-29-alpine-20250217104113
	baremetal-operator	mirantis.azurecr.io/bm/baremetal-operator:base-2-29-alpine-20250217104322
	bm-collective	mirantis.azurecr.io/bm/bm-collective:base-2-29-alpine-20250217104943
	cluster-api-provider-baremetal	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.42.9
	ironic	mirantis.azurecr.io/openstack/ironic:caracal-jammy-20250128120200
	ironic-inspector	mirantis.azurecr.io/openstack/ironic-inspector:caracal-jammy-20250128120200
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20240913123302
	kaas-ipam	mirantis.azurecr.io/core/kaas-ipam:1.42.9
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-202a68c-20250203183923
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.20-jammy-20241104184039
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20250217103755

Core artifacts¶

Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.42.9.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.42.9.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.42.9.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.42.9.tgz
	byo-provider ^Removed	n/a
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.42.9.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.42.9.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.42.9.tgz
	credentials-controller ^Deprecated	https://binary.mirantis.com/core/helm/credentials-controller-1.42.9.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.42.9.tgz
	host-os-modules-controller	https://binary.mirantis.com/core/helm/host-os-modules-controller-1.42.9.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.42.9.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.42.9.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.42.9.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.42.9.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.42.9.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.42.9.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.42.9.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.42.9.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.42.9.tgz
	openstack-provider ^Deprecated	https://binary.mirantis.com/core/helm/openstack-provider-1.42.9.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.42.9.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.42.9.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.42.9.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.42.9.tgz
	secret-controller	https://binary.mirantis.com/core/helm/secret-controller-1.42.9.tgz
	squid-proxy ^Removed	n/a
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.42.9.tgz
Docker images ^Updated	admission-controller	mirantis.azurecr.io/core/admission-controller:1.42.9
	agent-controller	mirantis.azurecr.io/core/agent-controller:1.42.9
	byo-cluster-api-controller ^Removed	n/a
	ceph-kcc-controller	mirantis.azurecr.io/core/ceph-kcc-controller:1.42.9
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-11
	configuration-collector	mirantis.azurecr.io/core/configuration-collector:1.42.9
	credentials-controller ^Deprecated	mirantis.azurecr.io/core/credentials-controller:1.42.9
	event-controller	mirantis.azurecr.io/core/event-controller:1.42.9
	frontend	mirantis.azurecr.io/core/frontend:1.42.9
	host-os-modules-controller	mirantis.azurecr.io/core/host-os-modules-controller:1.42.9
	iam-controller	mirantis.azurecr.io/core/iam-controller:1.42.9
	kaas-exporter	mirantis.azurecr.io/core/kaas-exporter:1.42.9
	kproxy	mirantis.azurecr.io/core/kproxy:1.42.9
	lcm-controller	mirantis.azurecr.io/core/lcm-controller:1.42.9
	license-controller	mirantis.azurecr.io/core/license-controller:1.42.9
	machinepool-controller	mirantis.azurecr.io/core/machinepool-controller:1.42.9
	mcc-cache-warmup	mirantis.azurecr.io/core/mcc-cache-warmup:1.42.9
	nginx	mirantis.azurecr.io/core/external/nginx:1.42.9
	openstack-cluster-api-controller ^Deprecated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.42.9
	portforward-controller	mirantis.azurecr.io/core/portforward-controller:1.42.9
	rbac-controller	mirantis.azurecr.io/core/rbac-controller:1.42.9
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-15
	release-controller	mirantis.azurecr.io/core/release-controller:1.42.9
	scope-controller	mirantis.azurecr.io/core/scope-controller:1.42.9
	secret-controller	mirantis.azurecr.io/core/secret-controller:1.42.9
	squid-proxy ^Removed	n/a
	user-controller	mirantis.azurecr.io/core/user-controller:1.42.9

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts ^Updated	iam	https://binary.mirantis.com/core/helm/iam-1.42.9.tgz
Docker images	kubectl	mirantis.azurecr.io/general/kubectl:20240926142019
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-ba8ada4-20240405150338
	mariadb ^Updated	mirantis.azurecr.io/general/mariadb:10.6.20-jammy-20241104184039
	mcc-keycloak ^Updated	mirantis.azurecr.io/iam/mcc-keycloak:25.0.6-20241114073807

Security notes¶

In total, since Container Cloud 2.28.5, in 2.29.0, 736 Common Vulnerabilities and Exposures (CVE) have been fixed: 125 of critical and 611 of high severity.

The table below includes the total numbers of addressed unique and common vulnerabilities and exposures (CVE) by product component since the 2.28.5 patch release. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	Critical	High	Total
Ceph	Unique	0	6	6
Ceph	Common	0	177	177
Kaas core	Unique	1	8	9
Kaas core	Common	88	229	317
StackLight	Unique	7	48	55
StackLight	Common	37	205	242

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK release notes 25.1: Security notes.

Update notes¶

This section describes the specific actions you as a cloud operator need to complete before or after your Container Cloud cluster update to the Cluster releases 17.4.0 or 16.4.0. For details on update impact and maintenance window planning, see MOSK Update notes.

Consider the information below as a supplement to the generic update procedures published in MOSK Operations Guide: Workflow and configuration of management cluster upgrade and MOSK Cluster update.

Pre-update actions¶

Update managed clusters to Ubuntu 22.04¶

In Container Cloud 2.29.0, the Cluster release update of the Ubuntu 20.04-based managed clusters becomes impossible, and Ubuntu 22.04 becomes the only supported version of the operating system. Therefore, ensure that every node of your managed clusters are running Ubuntu 22.04 to unblock managed cluster update in Container Cloud 2.29.0.

For the update procedure, refer to Mirantis OpenStack for Kubernetes documentation: Bare metal operations - Upgrade an operating system distribution.

Warning

Management cluster update to Container Cloud 2.29.1 will be blocked if at least one node of any related managed cluster is running Ubuntu 20.04.

Note

Back up custom Grafana dashboards on managed clusters¶

In Container Cloud 2.29.0, Grafana is updated to version 11 where the following deprecated Angular-based plugins are automatically migrated to the React-based ones:

Graph (old) -> Time Series
Singlestat -> Stat
Stat (old) -> Stat
Table (old) -> Table
Worldmap -> Geomap

This migration may corrupt custom Grafana dashboards that have Angular-based panels. Therefore, if you have such dashboards on managed clusters, back them up and manually upgrade Angular-based panels before updating to the Cluster release 17.4.0 to prevent custom appearance issues after plugin migration.

Note

All Grafana dashboards provided by StackLight are also migrated to React automatically. For the list of default dashboards, see MOSK Operations Guide: View Grafana dashboards.

Caution

For management clusters that are updated automatically, it is important to remove all Angular-based panels and prepare the backup of custom Grafana dashboards before Container Cloud 2.29.0 is released. For details, see Post update notes in 2.28.5 release notes. Otherwise, custom dashboards using Angular-based plugins may be corrupted and must be manually restored without a backup.

See also

Ensure that local-volume-provisioner is healthy¶

To ensure that local-volume-provisioner does not block cluster update and is healthy, either verify the cluster status in the Container Cloud web UI or run the following command:

kubectl -n kube-system get pods

Example of system response extract with healthy local-volume-provisioner:

local-volume-provisioner-dg98z 1/1 Running 0 66d

Example of system response extract with unhealthy local-volume-provisioner:

local-volume-provisioner-h5lrc   0/1   CrashLoopBackOff   33 (2m3s ago)   90m

If local-volume-provisioner is unhealthy, proceed with the workaround provided in the 50561 issue description.

Post-update actions¶

Start using BareMetalHostInventory instead of BareMetalHost¶

Container Cloud 2.29.0 introduces the BareMetalHostInventory resource that must be used instead of BareMetalHost for adding and modifying configuration of bare metal servers. Therefore, if you need to modify an existing or create a new configuration of a bare metal host, use BareMetalHostInventory.

Each BareMetalHostInventory object is synchronized with an automatically created BareMetalHost object, which is now used for internal purposes of the Container Cloud private API.

Caution

Any change in the BareMetalHost object will be overwitten by BareMetalHostInventory.

For any existing BareMetalHost object, a BareMetalHostInventory object is created automatically during cluster update.

Update passwords for custom Linux accounts¶

To match CIS Benchmark compliance checks for Ubuntu Linux 22.04 LTS v2.0.0 L1 Server, Container Cloud 2.29.0 introduces new password policies for local (Linux) user accounts. For details, see Improvements in the CIS Benchmark compliance for Ubuntu, MKE, and Docker.

The rules are applied automatically to all cluster nodes during cluster update. Therefore, if you use custom Linux accounts protected by passwords, do not plan any critical maintenance activities right after cluster upgrade as you may need to update Linux user passwords.

Note

By default, during cluster creation, mcc-user is created without a password with an option to add an SSH key.

Migrate container runtime from Docker to containerd¶

Container Cloud 2.29.0 introduces switching of the default container runtime from Docker to containerd on greenfield management and managed clusters.

Important

Container runtime migration involves machine cordoning and draining.

Note

If you have not upgraded the operating system distribution on your machines to Jammy yet, Mirantis recommends migrating machines from Docker to containerd on managed clusters together with distribution upgrade to minimize the maintenance window.

In this case, ensure that all cluster machines are updated at once during the same maintenance window to prevent machines from running different container runtimes.

Required action

MOSK Operations Guide: Migrate container runtime from Docker to containerd

See also

2.28.5¶

The Container Cloud patch release 2.28.5, which is based on the 2.28.0 major release, provides the following updates:

Support for the patch Cluster release 16.3.5.
Support for the patch Cluster release 17.3.5 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 24.3.2.
Support for Mirantis Kubernetes Engine 3.7.18 and Mirantis Container Runtime 23.0.15, which includes containerd 1.6.36.
Optional migration of container runtime from Docker to containerd.
Bare metal: update of Ubuntu mirror from ubuntu-2024-12-05-003900 to ubuntu-2025-01-08-003900 along with update of minor kernel version from 5.15.0-126-generic to 5.15.0-130-generic.
Security fixes for CVEs in images.

This patch release also supports the latest major Cluster releases 17.3.0 and 16.3.0. And it does not support greenfield deployments based on deprecated Cluster releases. Use the latest available Cluster release instead.

For main deliverables of the parent Container Cloud release of 2.28.5, refer to 2.28.0.

Update notes¶

This section describes the specific actions you as a cloud operator need to complete before or after your Container Cloud cluster update to the Cluster releases 17.3.5 or 16.3.5.

Consider the information below as a supplement to the generic update procedures published in MOSK Operations Guide: Automatic upgrade of a management cluster and Update to a patch version.

Post-update actions¶

Optional migration of container runtime from Docker to containerd¶

Since Container Cloud 2.28.4, Mirantis introduced an optional migration of container runtime from Docker to containerd, which is implemented for existing management and managed bare metal clusters. The use of containerd allows for better Kubernetes performance and component update without pod restart when applying fixes for CVEs. For the migration procedure, refer to MOSK Operations Guide: Migrate container runtime from Docker to containerd.

Note

Container runtime migration becomes mandatory in the scope of Container Cloud 2.29.x. Otherwise, the management cluster update to Container Cloud 2.30.0 will be blocked.

Note

In Containter Cloud 2.28.x series, the default container runtime remains Docker for greenfield deployments. Support for greenfield deployments based on containerd will be announced in one of the following releases.

Important

Container runtime migration involves machine cordoning and draining.

Note

In this case, ensure that all cluster machines are updated at once during the same maintenance window to prevent machines from running different container runtimes.

Back up custom Grafana dashboards¶

In Container Cloud 2.29.0, Grafana will be updated to version 11 where the following deprecated Angular-based plugins will be automatically migrated to the React-based ones:

Graph (old) -> Time Series
Singlestat -> Stat
Stat (old) -> Stat
Table (old) -> Table
Worldmap -> Geomap

This migration may corrupt custom Grafana dashboards that have Angular-based panels. Therefore, if you have such dashboards, back them up and manually upgrade Angular-based panels during the course of Container Cloud 2.28.x (Cluster releases 17.3.x and 16.3.x) to prevent custom appearance issues after plugin migration in Container Cloud 2.29.0 (Cluster releases 17.4.0 and 16.4.0).

Note

All Grafana dashboards provided by StackLight are also migrated to React automatically. For the list of default dashboards, see MOSK Operations Guide: View Grafana dashboards.

Warning

For management clusters that are updated automatically, it is important to prepare the backup before Container Cloud 2.29.0 is released. Otherwise, custom dashboards using Angular-based plugins may be corrupted.

For managed clusters, you can perform the backup after the Container Cloud 2.29.0 release date but before updating them to the Cluster release 17.4.0.

See also

Security notes¶

In total, since Container Cloud 2.28.4, 1 Common Vulnerability and Exposure (CVE) of high severity has been fixed in 2.28.5.

The table below includes the total numbers of addressed unique and common CVEs in images by product component since Container Cloud 2.28.4. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	Critical	High	Total
Kaas core	Unique	0	1	1
Kaas core	Common	0	1	1

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 24.3.2: Security notes.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.28.5 including the Cluster releases 16.3.5 and 17.3.5. For the known issues in the related MOSK release, see MOSK release notes 24.3.2: Known issues.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[47202] Inspection error on bare metal hosts after dnsmasq restart¶

Note

Moving forward, the workaround for this issue will be moved from Release Notes to MOSK Troubleshooting Guide: Inspection error on bare metal hosts after dnsmasq restart.

If the dnsmasq pod is restarted during the bootstrap of newly added nodes, those nodes may fail to undergo inspection. That can result in inspection error in the corresponding BareMetalHost objects.

The issue can occur when:

The dnsmasq pod was moved to another node.
DHCP subnets were changed, including addition or removal. In this case, the dhcpd container of the dnsmasq pod is restarted.

Caution

If changing or adding of DHCP subnets is required to bootstrap new nodes, wait after changing or adding DHCP subnets until the dnsmasq pod becomes ready, then create BareMetalHost objects.

To verify whether the nodes are affected:

Verify whether the BareMetalHost objects contain the inspection error:

kubectl get bmh -n <managed-cluster-namespace-name>

Example of system response:

NAME            STATE         CONSUMER        ONLINE   ERROR              AGE
test-master-1   provisioned   test-master-1   true                        9d
test-master-2   provisioned   test-master-2   true                        9d
test-master-3   provisioned   test-master-3   true                        9d
test-worker-1   provisioned   test-worker-1   true                        9d
test-worker-2   provisioned   test-worker-2   true                        9d
test-worker-3   inspecting                    true     inspection error   19h

Verify whether the dnsmasq pod was in Ready state when the inspection of the affected baremetal hosts (test-worker-3 in the example above) was started:

kubectl -n kaas get pod <dnsmasq-pod-name> -oyaml

Example of system response:

...
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-10-10T15:37:34Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2024-10-11T07:38:54Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2024-10-11T07:38:54Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2024-10-10T15:37:34Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://6dbcf2fc4b36ce4c549c9191ab01f72d0236c51d42947675302675e4bfaf4cdf
    image: docker-dev-kaas-virtual.artifactory-eu.mcp.mirantis.net/bm/baremetal-dnsmasq:base-2-28-alpine-20240812132650
    imageID: docker-dev-kaas-virtual.artifactory-eu.mcp.mirantis.net/bm/baremetal-dnsmasq@sha256:3dad3e278add18e69b2608e462691c4823942641a0f0e25e6811e703e3c23b3b
    lastState:
      terminated:
        containerID: containerd://816fcf079cd544acd74e312065de5b5ed4dbf1dc6159fefffff4f644b5e45987
        exitCode: 0
        finishedAt: "2024-10-11T07:38:35Z"
        reason: Completed
        startedAt: "2024-10-10T15:37:45Z"
    name: dhcpd
    ready: true
    restartCount: 2
    started: true
    state:
      running:
        startedAt: "2024-10-11T07:38:37Z"
  ...

In the system response above, the dhcpd container was not ready between "2024-10-11T07:38:35Z" and "2024-10-11T07:38:54Z".

Verify the affected baremetal host. For example:

kubectl get bmh -n managed-ns test-worker-3 -oyaml

Example of system response:

...
status:
  errorCount: 15
  errorMessage: Introspection timeout
  errorType: inspection error
  ...
  operationHistory:
    deprovision:
      end: null
      start: null
    inspect:
      end: null
      start: "2024-10-11T07:38:19Z"
    provision:
      end: null
      start: null
    register:
      end: "2024-10-11T07:38:19Z"
      start: "2024-10-11T07:37:25Z"

In the system response above, inspection was started at "2024-10-11T07:38:19Z", immediately before the period of the dhcpd container downtime. Therefore, this node is most likely affected by the issue.

Workaround

Reboot the node using the IPMI reset or cycle command.
If the node fails to boot, remove the failed BareMetalHost object and create it again:
1. Remove BareMetalHost object. For example:
```
kubectl delete bmh -n managed-ns test-worker-3
```
2. Verify that the BareMetalHost object is removed:
```
kubectl get bmh -n managed-ns test-worker-3
```
3. Create a BareMetalHost object from the template. For example:
```
kubectl create -f bmhc-test-worker-3.yaml
kubectl create -f bmh-test-worker-3.yaml
```

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

Ceph¶

[50566] Ceph upgrade is very slow during patch or major cluster update¶

Fixed in 2.29.3 (17.3.8, 16.3.8, and 16.4.3)

 Warning  Unhealthy  57s (x16 over 3m27s)  kubelet  Startup probe failed:
 ceph daemon health check failed with the following output:
> no valid command found; 10 closest matches:
> 0
> 1
> 2
> abort
> assert
> bluefs debug_inject_read_zeros
> bluefs files list
> bluefs stats
> bluestore bluefs device info [<alloc_size:int>]
> config diff
> admin_socket: invalid command

Workaround:

Complete the following steps during every patch or major cluster update of the Cluster releases 17.2.x, 17.3.x, and 17.4.x (until Ceph 18.2.5 becomes supported):

Plan extra time in the maintenance window for the patch cluster update.

Slow starts will still impact the update procedure, but after completing the following step, the recovery process noticeably shortens without affecting the overall cluster state and data responsiveness.
Select one of the following options:
- Before the cluster update, set the noout flag:
```
ceph osd set noout
```
  Once the Ceph OSDs image upgrade is done, unset the flag:
```
ceph osd unset noout
```
- Monitor the Ceph OSDs image upgrade. If the symptoms of slow start appear, set the noout flag as soon as possible. Once the Ceph OSDs image upgrade is done, unset the flag.

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

LCM¶

[39437] Failure to replace a master node on a Container Cloud cluster¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

During the replacement of a master node on a cluster of any type, the process may get stuck with Kubelet's NodeReady condition is Unknown in the machine status on the remaining master nodes.

As a workaround, log in on the affected node and run the following command:

docker restart ucp-kubelet

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

StackLight¶

[44193] OpenSearch reaches 85% disk usage watermark affecting the cluster state¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

On High Availability (HA) clusters that use Local Volume Provisioner (LVP), Prometheus and OpenSearch from StackLight may share the same pool of storage. In such configuration, OpenSearch may approach the 85% disk usage watermark due to the combined storage allocation and usage patterns set by the Persistent Volume Claim (PVC) size parameters for Prometheus and OpenSearch, which consume storage the most.

When the 85% threshold is reached, the affected node is transitioned to the read-only state, preventing shard allocation and causing the OpenSearch cluster state to transition to Warning (Yellow) or Critical (Red).

Caution

The issue and the provided workaround apply only for clusters on which OpenSearch and Prometheus utilize the same storage pool.

To verify that the cluster is affected:

Verify the result of the following formula:
```
0.8 × OpenSearch_PVC_Size_GB + Prometheus_PVC_Size_GB > 0.85 × Total_Storage_Capacity_GB
```
In the formula, define the following values:
OpenSearch_PVC_Size_GB
Derived from .values.elasticsearch.persistentVolumeUsableStorageSizeGB, defaulting to .values.elasticsearch.persistentVolumeClaimSize if unspecified. To obtain the OpenSearch PVC size:
kubectl -n <namespaceName> get cluster <clusterName> -o yaml |\ yq '.spec.providerSpec.value.helmReleases[] | select(.name == "stacklight") | .values.elasticsearch.persistentVolumeClaimSize '
Example of system response:
10000Gi
Prometheus_PVC_Size_GB
Sourced from .values.prometheusServer.persistentVolumeClaimSize. To obtain the Prometheus PVC size:
kubectl -n <namespaceName> get cluster <clusterName> -o yaml |\ yq '.spec.providerSpec.value.helmReleases[] | select(.name == "stacklight") | .values.prometheusServer.persistentVolumeClaimSize '
Example of system response:
4000Gi
Total_Storage_Capacity_GB
Total capacity of the OpenSearch PVCs. For LVP, the capacity of the storage pool. To obtain the total capacity:
kubectl get pvc -n stacklight -l app=opensearch-master \ -o custom-columns=NAME:.metadata.name,CAPACITY:.status.capacity.storage
The system response contains multiple outputs, one per opensearch-master node. Select the capacity for the affected node.
Note

Convert the values to GB if they are set in different units.

If the formula result is positive, it is an early indication that the cluster is affected.
Verify whether the OpenSearchClusterStatusWarning or OpenSearchClusterStatusCritical alert is firing. And if so, verify the following:
1. Log in to the OpenSearch web UI.
2. In Management -> Dev Tools, run the following command:
```
GET _cluster/allocation/explain
```
  The following system response indicates that the corresponding node is affected:
```
"explanation": "the node is above the low watermark cluster setting \
[cluster.routing.allocation.disk.watermark.low=85%], using more disk space \
than the maximum allowed [85.0%], actual free: [xx.xxx%]"
```
  Note
  
  The system response may contain even higher watermark percent than 85.0%, depending on the case.

Workaround:

Warning

The workaround implies adjustement of the retention threshold for OpenSearch. And depending on the new threshold, some old logs will be deleted.

Adjust or set .values.elasticsearch.persistentVolumeUsableStorageSizeGB to a lower value for the affection check formula to be non-positive. For configuration details, see MOSK Operations Guide: StackLight configuration parameters - OpenSearch.

Mirantis also recommends reserving some space for other PVCs using storage from the pool. Use the following formula to calculate the required space:
```
persistentVolumeUsableStorageSizeGB =
0.84 × ((1 - Reserved_Percentage - Filesystem_Reserve) ×
Total_Storage_Capacity_GB - Prometheus_PVC_Size_GB) /
0.8
```
In the formula, define the following values:
Reserved_Percentage
A user-defined variable that specifies what percentage of the total storage capacity should not be used by OpenSearch or Prometheus. This is used to reserve space for other components. It should be expressed as a decimal. For example, for 5% of reservation, Reserved_Percentage is 0.05. Mirantis recommends using 0.05 as a starting point.

Filesystem_Reserve
Percentage to deduct for filesystems that may reserve some portion of the available storage, which is marked as occupied. For example, for EXT4, it is 5% by default, so the value must be 0.05.

Prometheus_PVC_Size_GB
Sourced from .values.prometheusServer.persistentVolumeClaimSize.

Total_Storage_Capacity_GB
Total capacity of the OpenSearch PVCs. For LVP, the capacity of the storage pool. To obtain the total capacity:
kubectl get pvc -n stacklight -l app=opensearch-master \ -o custom-columns=NAME:.metadata.name,CAPACITY:.status.capacity.storage
The system response contains multiple outputs, one per opensearch-master node. Select the capacity for the affected node.
Note

Convert the values to GB if they are set in different units.

Calculation of above formula provides a maximum safe storage to allocate for .values.elasticsearch.persistentVolumeUsableStorageSizeGB. Use this formula as a reference for setting .values.elasticsearch.persistentVolumeUsableStorageSizeGB on a cluster.
Wait up to 15-20 mins for OpenSearch to perform the cleaning.
Verify that the cluster is not affected anymore using the procedure above.

Container Cloud web UI¶

[50181] Failure to deploy a compact cluster¶

To work around the issue, manually add the required labels using CLI. Once done, the cluster deployment resumes.

[50168] Inability to use a new project right after creation¶

A newly created project does not display all available tabs in the Container Cloud web UI and contains different access denied errors during first five minutes after creation.

To work around the issue, refresh the browser in five minutes after the project creation.

Artifacts¶

This section lists the artifacts of components included in the Container Cloud patch release 2.28.5. For artifacts of the Cluster releases introduced in 2.28.5, see patch Cluster releases 17.3.5 and 16.3.5.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries	ironic-python-agent.initramfs ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-antelope-jammy-debug-20250108133235
	ironic-python-agent.kernel ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-antelope-jammy-debug-20250108133235
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-167-e7a55fd.tgz
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.41.28.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.41.28.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.41.28.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.41.28.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.41.28.tgz
Docker images	ambassador ^Updated	mirantis.azurecr.io/core/external/nginx:1.41.28
	baremetal-dnsmasq	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-28-alpine-20241022121257
	baremetal-operator	mirantis.azurecr.io/bm/baremetal-operator:base-2-28-alpine-20241217153430
	bm-collective	mirantis.azurecr.io/bm/bm-collective:base-2-28-alpine-20241217153957
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.41.28
	ironic	mirantis.azurecr.io/openstack/ironic:antelope-jammy-20241128095555
	ironic-inspector	mirantis.azurecr.io/openstack/ironic-inspector:antelope-jammy-20241128095555
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20240913123302
	kaas-ipam	mirantis.azurecr.io/bm/kaas-ipam:base-2-28-alpine-20241217153549
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-34a4f54-20240910081335
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.17-jammy-20240927170336
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20241022120929

Core artifacts¶

Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.41.28.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.41.28.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.41.28.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.41.28.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.41.28.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.41.28.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.41.28.tgz
	credentials-controller ^Deprecated	https://binary.mirantis.com/core/helm/credentials-controller-1.41.28.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.41.28.tgz
	host-os-modules-controller	https://binary.mirantis.com/core/helm/host-os-modules-controller-1.41.28.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.41.28.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.41.28.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.41.28.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.41.28.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.41.28.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.41.28.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.41.28.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.41.28.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.41.28.tgz
	openstack-provider ^Deprecated	https://binary.mirantis.com/core/helm/openstack-provider-1.41.28.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.41.28.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.41.28.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.41.28.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.41.28.tgz
	secret-controller	https://binary.mirantis.com/core/helm/secret-controller-1.41.28.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.41.28.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.41.28
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.41.28
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.41.28
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-9
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.41.28
	credentials-controller ^Deprecated	mirantis.azurecr.io/core/credentials-controller:1.41.28
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.41.28
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.41.28
	host-os-modules-controller ^Updated	mirantis.azurecr.io/core/host-os-modules-controller:1.41.28
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.41.28
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.41.28
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.41.28
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.41.28
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.41.28
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.41.28
	mcc-cache-warmup ^Updated	mirantis.azurecr.io/core/mcc-cache-warmup:1.41.28
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.41.28
	openstack-cluster-api-controller ^Deprecated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.41.28
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.41.28
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.41.28
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-14
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.41.28
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.41.28
	secret-controller ^Updated	mirantis.azurecr.io/core/secret-controller:1.41.28
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.41.28

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts ^Updated	iam	https://binary.mirantis.com/core/helm/iam-1.41.28.tgz
Docker images	kubectl	mirantis.azurecr.io/general/kubectl:20240926142019
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-ba8ada4-20240405150338
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.17-focal-20240909113408
	mcc-keycloak	mirantis.azurecr.io/iam/mcc-keycloak:25.0.6-20241114073807

2.28.4¶

The Container Cloud patch release 2.28.4, which is based on the 2.28.0 major release, provides the following updates:

Support for the patch Cluster release 16.3.4.
Support for the patch Cluster release 17.3.4 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 24.3.1.
Support for Mirantis Kubernetes Engine to 3.7.17 and Mirantis Container Runtime 23.0.15, which includes containerd 1.6.36.
Optional migration of container runtime from Docker to containerd.
Bare metal: update of Ubuntu mirror from ubuntu-2024-11-18-003900 to ubuntu-2024-12-05-003900 along with update of minor kernel version from 5.15.0-125-generic to 5.15.0-126-generic.
Security fixes for CVEs in images.
OpenStack provider: suspension of support for cluster deployment and update. For details, see Deprecation notes.

For main deliverables of the parent Container Cloud release of 2.28.4, refer to 2.28.0.

Update notes¶

This section describes the specific actions you as a cloud operator need to complete before or after your Container Cloud cluster update to the Cluster releases 17.3.4 or 16.3.4.

Important

For MOSK deployments, although MOSK 24.3.1 is classified as a patch release, as a cloud operator, you will be performing a major update regardless of the upgrade path: whether you are upgrading from patch 24.2.5 or major version 24.3. For details, see MOSK 24.3.1 release notes: Update notes.

Consider the information below as a supplement to the generic update procedures published in MOSK Operations Guide: Automatic upgrade of a management cluster and Update to a patch version.

Post-update actions¶

Optional migration of container runtime from Docker to containerd¶

Container Cloud 2.28.4 introduces an optional migration of container runtime from Docker to containerd, which is implemented for existing management and managed bare metal clusters. The use of containerd allows for better Kubernetes performance and component update without pod restart when applying fixes for CVEs. For the migration procedure, refer to MOSK Operations Guide: Migrate container runtime from Docker to containerd.

Note

Container runtime migration becomes mandatory in the scope of Container Cloud 2.29.x. Otherwise, the management cluster update to Container Cloud 2.30.0 will be blocked.

Note

Important

Container runtime migration involves machine cordoning and draining.

Note

In this case, ensure that all cluster machines are updated at once during the same maintenance window to prevent machines from running different container runtimes.

Back up custom Grafana dashboards¶

In Container Cloud 2.29.0, Grafana will be updated to version 11 where the following deprecated Angular-based plugins will be automatically migrated to the React-based ones:

Graph (old) -> Time Series
Singlestat -> Stat
Stat (old) -> Stat
Table (old) -> Table
Worldmap -> Geomap

Note

All Grafana dashboards provided by StackLight are also migrated to React automatically. For the list of default dashboards, see MOSK Operations Guide: View Grafana dashboards.

Warning

For managed clusters, you can perform the backup after the Container Cloud 2.29.0 release date but before updating them to the Cluster release 17.4.0.

See also

Security notes¶

In total, since Container Cloud 2.28.3, 158 Common Vulnerabilities and Exposures (CVE) have been fixed in 2.28.4: 10 of critical and 148 of high severity.

The table below includes the total numbers of addressed unique and common CVEs in images by product component since Container Cloud 2.28.3. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	Critical	High	Total
Ceph	Unique	0	3	3
Ceph	Common	0	7	7
Kaas core	Unique	1	18	19
Kaas core	Common	4	92	96
StackLight	Unique	3	19	22
StackLight	Common	6	49	55

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 24.3.1: Security notes.

Addressed issues¶

The following issues have been addressed in the Container Cloud patch release 2.28.4 along with the patch Cluster releases 16.3.4 and 17.3.4:

[30294] [LCM] Fixed the issue that prevented replacement of a manager machine during the calico-node Pod start on a new node that has the same IP address as the node being replaced.
[5782] [LCM] Fixed the issue that prevented deployment of a manager machine during node replacement.
[5568] [LCM] Fixed the issue that prevented cleaning of resources by the calico-kube-controllers Pod during unsafe or forced deletion of a manager machine.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.28.4 including the Cluster releases 16.3.4 and 17.3.4. For the known issues in the related MOSK release, see MOSK release notes 24.3.1: Known issues.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[47202] Inspection error on bare metal hosts after dnsmasq restart¶

Note

Moving forward, the workaround for this issue will be moved from Release Notes to MOSK Troubleshooting Guide: Inspection error on bare metal hosts after dnsmasq restart.

The issue can occur when:

The dnsmasq pod was moved to another node.
DHCP subnets were changed, including addition or removal. In this case, the dhcpd container of the dnsmasq pod is restarted.

Caution

If changing or adding of DHCP subnets is required to bootstrap new nodes, wait after changing or adding DHCP subnets until the dnsmasq pod becomes ready, then create BareMetalHost objects.

To verify whether the nodes are affected:

Verify whether the BareMetalHost objects contain the inspection error:

kubectl get bmh -n <managed-cluster-namespace-name>

Example of system response:

NAME            STATE         CONSUMER        ONLINE   ERROR              AGE
test-master-1   provisioned   test-master-1   true                        9d
test-master-2   provisioned   test-master-2   true                        9d
test-master-3   provisioned   test-master-3   true                        9d
test-worker-1   provisioned   test-worker-1   true                        9d
test-worker-2   provisioned   test-worker-2   true                        9d
test-worker-3   inspecting                    true     inspection error   19h

Verify whether the dnsmasq pod was in Ready state when the inspection of the affected baremetal hosts (test-worker-3 in the example above) was started:

kubectl -n kaas get pod <dnsmasq-pod-name> -oyaml

Example of system response:

...
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-10-10T15:37:34Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2024-10-11T07:38:54Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2024-10-11T07:38:54Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2024-10-10T15:37:34Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://6dbcf2fc4b36ce4c549c9191ab01f72d0236c51d42947675302675e4bfaf4cdf
    image: docker-dev-kaas-virtual.artifactory-eu.mcp.mirantis.net/bm/baremetal-dnsmasq:base-2-28-alpine-20240812132650
    imageID: docker-dev-kaas-virtual.artifactory-eu.mcp.mirantis.net/bm/baremetal-dnsmasq@sha256:3dad3e278add18e69b2608e462691c4823942641a0f0e25e6811e703e3c23b3b
    lastState:
      terminated:
        containerID: containerd://816fcf079cd544acd74e312065de5b5ed4dbf1dc6159fefffff4f644b5e45987
        exitCode: 0
        finishedAt: "2024-10-11T07:38:35Z"
        reason: Completed
        startedAt: "2024-10-10T15:37:45Z"
    name: dhcpd
    ready: true
    restartCount: 2
    started: true
    state:
      running:
        startedAt: "2024-10-11T07:38:37Z"
  ...

In the system response above, the dhcpd container was not ready between "2024-10-11T07:38:35Z" and "2024-10-11T07:38:54Z".

Verify the affected baremetal host. For example:

kubectl get bmh -n managed-ns test-worker-3 -oyaml

Example of system response:

...
status:
  errorCount: 15
  errorMessage: Introspection timeout
  errorType: inspection error
  ...
  operationHistory:
    deprovision:
      end: null
      start: null
    inspect:
      end: null
      start: "2024-10-11T07:38:19Z"
    provision:
      end: null
      start: null
    register:
      end: "2024-10-11T07:38:19Z"
      start: "2024-10-11T07:37:25Z"

Workaround

Reboot the node using the IPMI reset or cycle command.
If the node fails to boot, remove the failed BareMetalHost object and create it again:
1. Remove BareMetalHost object. For example:
```
kubectl delete bmh -n managed-ns test-worker-3
```
2. Verify that the BareMetalHost object is removed:
```
kubectl get bmh -n managed-ns test-worker-3
```
3. Create a BareMetalHost object from the template. For example:
```
kubectl create -f bmhc-test-worker-3.yaml
kubectl create -f bmh-test-worker-3.yaml
```

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

Ceph¶

[50566] Ceph upgrade is very slow during patch or major cluster update¶

Fixed in 2.29.3 (17.3.8, 16.3.8, and 16.4.3)

 Warning  Unhealthy  57s (x16 over 3m27s)  kubelet  Startup probe failed:
 ceph daemon health check failed with the following output:
> no valid command found; 10 closest matches:
> 0
> 1
> 2
> abort
> assert
> bluefs debug_inject_read_zeros
> bluefs files list
> bluefs stats
> bluestore bluefs device info [<alloc_size:int>]
> config diff
> admin_socket: invalid command

Workaround:

Complete the following steps during every patch or major cluster update of the Cluster releases 17.2.x, 17.3.x, and 17.4.x (until Ceph 18.2.5 becomes supported):

Plan extra time in the maintenance window for the patch cluster update.

Slow starts will still impact the update procedure, but after completing the following step, the recovery process noticeably shortens without affecting the overall cluster state and data responsiveness.
Select one of the following options:
- Before the cluster update, set the noout flag:
```
ceph osd set noout
```
  Once the Ceph OSDs image upgrade is done, unset the flag:
```
ceph osd unset noout
```
- Monitor the Ceph OSDs image upgrade. If the symptoms of slow start appear, set the noout flag as soon as possible. Once the Ceph OSDs image upgrade is done, unset the flag.

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

LCM¶

[39437] Failure to replace a master node on a Container Cloud cluster¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

During the replacement of a master node on a cluster of any type, the process may get stuck with Kubelet's NodeReady condition is Unknown in the machine status on the remaining master nodes.

As a workaround, log in on the affected node and run the following command:

docker restart ucp-kubelet

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

StackLight¶

[44193] OpenSearch reaches 85% disk usage watermark affecting the cluster state¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

Caution

The issue and the provided workaround apply only for clusters on which OpenSearch and Prometheus utilize the same storage pool.

To verify that the cluster is affected:

Verify the result of the following formula:
```
0.8 × OpenSearch_PVC_Size_GB + Prometheus_PVC_Size_GB > 0.85 × Total_Storage_Capacity_GB
```
In the formula, define the following values:
OpenSearch_PVC_Size_GB
Derived from .values.elasticsearch.persistentVolumeUsableStorageSizeGB, defaulting to .values.elasticsearch.persistentVolumeClaimSize if unspecified. To obtain the OpenSearch PVC size:
kubectl -n <namespaceName> get cluster <clusterName> -o yaml |\ yq '.spec.providerSpec.value.helmReleases[] | select(.name == "stacklight") | .values.elasticsearch.persistentVolumeClaimSize '
Example of system response:
10000Gi
Prometheus_PVC_Size_GB
Sourced from .values.prometheusServer.persistentVolumeClaimSize. To obtain the Prometheus PVC size:
kubectl -n <namespaceName> get cluster <clusterName> -o yaml |\ yq '.spec.providerSpec.value.helmReleases[] | select(.name == "stacklight") | .values.prometheusServer.persistentVolumeClaimSize '
Example of system response:
4000Gi
Total_Storage_Capacity_GB
Total capacity of the OpenSearch PVCs. For LVP, the capacity of the storage pool. To obtain the total capacity:
kubectl get pvc -n stacklight -l app=opensearch-master \ -o custom-columns=NAME:.metadata.name,CAPACITY:.status.capacity.storage
The system response contains multiple outputs, one per opensearch-master node. Select the capacity for the affected node.
Note

Convert the values to GB if they are set in different units.

If the formula result is positive, it is an early indication that the cluster is affected.
Verify whether the OpenSearchClusterStatusWarning or OpenSearchClusterStatusCritical alert is firing. And if so, verify the following:
1. Log in to the OpenSearch web UI.
2. In Management -> Dev Tools, run the following command:
```
GET _cluster/allocation/explain
```
  The following system response indicates that the corresponding node is affected:
```
"explanation": "the node is above the low watermark cluster setting \
[cluster.routing.allocation.disk.watermark.low=85%], using more disk space \
than the maximum allowed [85.0%], actual free: [xx.xxx%]"
```
  Note
  
  The system response may contain even higher watermark percent than 85.0%, depending on the case.

Workaround:

Warning

The workaround implies adjustement of the retention threshold for OpenSearch. And depending on the new threshold, some old logs will be deleted.

Adjust or set .values.elasticsearch.persistentVolumeUsableStorageSizeGB to a lower value for the affection check formula to be non-positive. For configuration details, see MOSK Operations Guide: StackLight configuration parameters - OpenSearch.

Mirantis also recommends reserving some space for other PVCs using storage from the pool. Use the following formula to calculate the required space:
```
persistentVolumeUsableStorageSizeGB =
0.84 × ((1 - Reserved_Percentage - Filesystem_Reserve) ×
Total_Storage_Capacity_GB - Prometheus_PVC_Size_GB) /
0.8
```
In the formula, define the following values:
Reserved_Percentage
A user-defined variable that specifies what percentage of the total storage capacity should not be used by OpenSearch or Prometheus. This is used to reserve space for other components. It should be expressed as a decimal. For example, for 5% of reservation, Reserved_Percentage is 0.05. Mirantis recommends using 0.05 as a starting point.

Filesystem_Reserve
Percentage to deduct for filesystems that may reserve some portion of the available storage, which is marked as occupied. For example, for EXT4, it is 5% by default, so the value must be 0.05.

Prometheus_PVC_Size_GB
Sourced from .values.prometheusServer.persistentVolumeClaimSize.

Total_Storage_Capacity_GB
Total capacity of the OpenSearch PVCs. For LVP, the capacity of the storage pool. To obtain the total capacity:
kubectl get pvc -n stacklight -l app=opensearch-master \ -o custom-columns=NAME:.metadata.name,CAPACITY:.status.capacity.storage
The system response contains multiple outputs, one per opensearch-master node. Select the capacity for the affected node.
Note

Convert the values to GB if they are set in different units.

Calculation of above formula provides a maximum safe storage to allocate for .values.elasticsearch.persistentVolumeUsableStorageSizeGB. Use this formula as a reference for setting .values.elasticsearch.persistentVolumeUsableStorageSizeGB on a cluster.
Wait up to 15-20 mins for OpenSearch to perform the cleaning.
Verify that the cluster is not affected anymore using the procedure above.

Container Cloud web UI¶

[50181] Failure to deploy a compact cluster¶

To work around the issue, manually add the required labels using CLI. Once done, the cluster deployment resumes.

[50168] Inability to use a new project right after creation¶

A newly created project does not display all available tabs in the Container Cloud web UI and contains different access denied errors during first five minutes after creation.

To work around the issue, refresh the browser in five minutes after the project creation.

Artifacts¶

This section lists the artifacts of components included in the Container Cloud patch release 2.28.4. For artifacts of the Cluster releases introduced in 2.28.4, see patch Cluster releases 17.3.4 and 16.3.4.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries	ironic-python-agent.initramfs ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-antelope-jammy-debug-20241205172311
	ironic-python-agent.kernel ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-antelope-jammy-debug-20241205172311
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-167-e7a55fd.tgz
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.41.26.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.41.26.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.41.26.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.41.26.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.41.26.tgz
Docker images	ambassador ^Updated	mirantis.azurecr.io/core/external/nginx:1.41.26
	baremetal-dnsmasq	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-28-alpine-20241022121257
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-2-28-alpine-20241217153430
	bm-collective ^Updated	mirantis.azurecr.io/bm/bm-collective:base-2-28-alpine-20241217153957
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.41.26
	ironic	mirantis.azurecr.io/openstack/ironic:antelope-jammy-20241128095555
	ironic-inspector	mirantis.azurecr.io/openstack/ironic-inspector:antelope-jammy-20241128095555
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20240913123302
	kaas-ipam ^Updated	mirantis.azurecr.io/bm/kaas-ipam:base-2-28-alpine-20241217153549
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-34a4f54-20240910081335
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.17-jammy-20240927170336
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20241022120929

Core artifacts¶

Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.41.26.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.41.26.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.41.26.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.41.26.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.41.26.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.41.26.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.41.26.tgz
	credentials-controller	https://binary.mirantis.com/core/helm/credentials-controller-1.41.26.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.41.26.tgz
	host-os-modules-controller	https://binary.mirantis.com/core/helm/host-os-modules-controller-1.41.26.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.41.26.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.41.26.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.41.26.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.41.26.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.41.26.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.41.26.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.41.26.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.41.26.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.41.26.tgz
	openstack-provider ^Deprecated	https://binary.mirantis.com/core/helm/openstack-provider-1.41.26.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.41.26.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.41.26.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.41.26.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.41.26.tgz
	secret-controller	https://binary.mirantis.com/core/helm/secret-controller-1.41.26.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.41.26.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.41.26.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.41.26
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.41.26
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.41.26
	cert-manager-controller ^Updated	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-9
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.41.26
	credentials-controller ^Updated	mirantis.azurecr.io/core/credentials-controller:1.41.26
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.41.26
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.41.26
	host-os-modules-controller ^Updated	mirantis.azurecr.io/core/host-os-modules-controller:1.41.26
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.41.26
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.41.26
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.41.26
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.41.26
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.41.26
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.41.26
	mcc-cache-warmup ^Updated	mirantis.azurecr.io/core/mcc-cache-warmup:1.41.26
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.41.26
	openstack-cluster-api-controller ^Deprecated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.41.26
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.41.26
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.41.26
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-14
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.41.26
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.41.26
	secret-controller ^Updated	mirantis.azurecr.io/core/secret-controller:1.41.26
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.41.26

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts ^Updated	iam	https://binary.mirantis.com/core/helm/iam-1.41.26.tgz
Docker images	kubectl	mirantis.azurecr.io/general/kubectl:20240926142019
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-ba8ada4-20240405150338
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.17-focal-20240909113408
	mcc-keycloak	mirantis.azurecr.io/iam/mcc-keycloak:25.0.6-20241114073807

Releases delivered in 2024¶

This section contains historical information on the unsupported Container Cloud releases delivered in 2024. For the latest supported Container Cloud release, see Container Cloud releases.

Unsupported Container Cloud releases 2024¶
Version	Release date	Summary
2.28.3	Dec 09, 2024	Container Cloud 2.28.3 is the third patch release of the 2.28.x release series that introduces the following updates: Support for the patch Cluster release 16.3.3. Support for the patch Cluster releases 16.2.7 and 17.2.7 that represents MOSK patch release 24.2.5. Bare metal: update of Ubuntu mirror to ubuntu-2024-11-18-003900 along with update of minor kernel version to 5.15.0-125-generic. Security fixes for CVEs in images.
2.28.2	Nov 18, 2024	Container Cloud 2.28.2 is the second patch release of the 2.28.x release series that introduces the following updates: Support for the patch Cluster release 16.3.2. Support for the patch Cluster releases 16.2.6 and 17.2.6 that represents MOSK patch release 24.2.4. Support for MKE 3.7.16. Bare metal: update of Ubuntu mirror to ubuntu-2024-10-28-012906 along with update of minor kernel version to 5.15.0-124-generic. Security fixes for CVEs in images.
2.28.1	Oct 30, 2024	Container Cloud 2.28.1 is the first patch release of the 2.28.x release series that introduces the following updates: Support for the patch Cluster release 16.3.1. Support for the patch Cluster releases 16.2.5 and 17.2.5 that represents MOSK patch release 24.2.3. Support for MKE 3.7.15. Bare metal: update of Ubuntu mirror to ubuntu-2024-10-14-013948 along with update of minor kernel version to 5.15.0-122-generic. Security fixes for CVEs in images.
2.28.0	Oct 16, 2024	General availability for Ubuntu 22.04 on MOSK clusters Improvements in the CIS Benchmark compliance for Ubuntu Linux 22.04 LTS v2.0.0 L1 Server Support for MKE 3.7.12 on clusters following the major update path Support for MCR 23.0.14 Update group for controller nodes Reboot of machines using update groups Amendments for the `ClusterUpdatePlan` object Refactoring of delayed auto-update of a management cluster Self-diagnostics for management and managed clusters Configuration of groups in auditd Container Cloud web UI enhancements for the bare metal provider Day-2 operations for bare metal: Updating modules Configuration enhancements for modules StackLight: Monitoring of LCM issues Refactoring of StackLight expiration alerts Documentation enhancements
2.27.4	Sep 16, 2024	Container Cloud 2.27.4 is the fourth patch release of the 2.27.x release series that introduces the following updates: Support for the patch Cluster releases 16.2.4 and 17.2.4 that represents MOSK patch release 24.2.2. Bare metal: update of Ubuntu mirror to ubuntu-2024-08-21-014714 along with update of minor kernel version to 5.15.0-119-generic. Security fixes for CVEs in images.
2.27.3	Aug 27, 2024	Container Cloud 2.27.3 is the third patch release of the 2.27.x release series that introduces the following updates: Support for the patch Cluster releases 16.2.3 and 17.2.3 that represents MOSK patch release 24.2.1. Support for MKE 3.7.12. Improvements in the MKE benchmark compliance (control ID 5.1.5). Bare metal: update of Ubuntu mirror to ubuntu-2024-08-06-014502 along with update of minor kernel version to 5.15.0-117-generic. VMware vSphere: suspension of support for cluster deployment, update, and attachment. Security fixes for CVEs in images.
2.27.2	Aug 05, 2024	Container Cloud 2.27.2 is the second patch release of the 2.27.x release series that introduces the following updates: Support for the patch Cluster release 16.2.2. Support for the patch Cluster releases 16.1.7 and 17.1.7 that represents MOSK patch release 24.1.7. Support for MKE 3.7.11. Bare metal: update of Ubuntu mirror to ubuntu-2024-07-16-014744 along with update of the minor kernel version to 5.15.0-116-generic (Cluster release 16.2.2). Security fixes for CVEs in images.
2.27.1	Jul 16, 2024	Container Cloud 2.27.1 is the first patch release of the 2.27.x release series that introduces the following updates: Support for the patch Cluster release 16.2.1. Support for the patch Cluster releases 16.1.6 and 17.1.6 that represents MOSK patch release 24.1.6. Support for MKE 3.7.10. Support for `docker-ee-cli` 23.0.13 in MCR 23.0.11 to fix several CVEs. Bare metal: update of Ubuntu mirror to ubuntu-2024-06-27-095142 along with update of minor kernel version to 5.15.0-113-generic. Security fixes for CVEs in images. Bug fixes.
2.27.0	Jul 02, 2024	MKE: MKE 3.7.8 for clusters that follow major update path Improvements in the MKE benchmark compliance Bare metal: General availability for Ubuntu 22.04 on bare metal clusters Improvements in the day-2 management API for bare metal clusters Optimization of strict filtering for devices on bare metal clusters Deprecation of `SubnetPool` and `MetalLBConfigTemplate` objects LCM: The `ClusterUpdatePlan` object for a granular cluster update Update groups for worker machines LCM Agent heartbeats Handling secret leftovers using `secret-controller` MariaDB backup for bare metal and vSphere providers Ceph: Automatic upgrade from Quincy to Reef Support for Rook v1.13 Setting a configuration section for Rook parameters StackLight: Monitoring of I/O errors in kernel logs S.M.A.R.T. metrics for creating alert rules on bare metal clusters Improvements for `OpenSearch` and `OpenSearch Indices` Grafana dashboards Removal of `grafana-image-renderer`
2.26.5	June 18, 2024	Container Cloud 2.26.5 is the fifth patch release of the 2.26.x and MOSK 24.1.x release series that introduces the following updates: Support for the patch Cluster releases 16.1.5 and 17.1.5 that represents MOSK patch release 24.1.5. Bare metal: update of Ubuntu mirror to 20.04~20240517090228 along with update of minor kernel version to 5.15.0-107-generic. Security fixes for CVEs in images. Bug fixes.
2.26.4	May 20, 2024	Container Cloud 2.26.4 is the fourth patch release of the 2.26.x and MOSK 24.1.x release series that introduces the following updates: Support for the patch Cluster releases 16.1.4 and 17.1.4 that represents MOSK patch release 24.1.4. Support for MKE 3.7.8. Bare metal: update of Ubuntu mirror to 20.04~20240502102020 along with update of minor kernel version to 5.15.0-105-generic. Security fixes for CVEs in images. Bug fixes.
2.26.3	Apr 29, 2024	Container Cloud 2.26.3 is the third patch release of the 2.26.x and MOSK 24.1.x release series that introduces the following updates: Support for the patch Cluster releases 16.1.3 and 17.1.3 that represents MOSK patch release 24.1.3. Support for MKE 3.7.7. Bare metal: update of Ubuntu mirror to 20.04~20240411171541 along with update of minor kernel version to 5.15.0-102-generic. Security fixes for CVEs in images. Bug fixes.
2.26.2	Apr 08, 2024	Container Cloud 2.26.2 is the second patch release of the 2.26.x and MOSK 24.1.x release series that introduces the following updates: Support for the patch Cluster releases 16.1.2 and 17.1.2 that represents MOSK patch release 24.1.2. Support for MKE 3.7.6. Support for `docker-ee-cli` 23.0.10 in MCR 23.0.9 to fix the several CVEs. Bare metal: update of Ubuntu mirror to 20.04~20240324172903 along with update of minor kernel version to 5.15.0-101-generic. Security fixes for CVEs in images.
2.26.1	Mar 20, 2024	Container Cloud 2.26.1 is the first patch release of the 2.26.x and MOSK 24.1.x release series that introduces the following updates: Support for the patch Cluster releases 16.1.1 and 17.1.1 that represents MOSK patch release 24.1.1. Support for MKE 3.7.6. Security fixes for CVEs in images.
2.26.0	Mar 04, 2024	LCM: Pre-update inspection of pinned product artifacts in a `Cluster` object Disablement of worker machines on managed clusters Health monitoring of cluster LCM operations Support for MKE 3.7.5 and MCR 23.0.9 Security: Support for Kubernetes auditing and profiling on management clusters Policy Controller for validating pod image signatures Configuring trusted certificates for Keycloak Bare metal: Day-2 management API for bare metal clusters Strict filtering for devices on bare metal clusters Dynamic IP allocation for faster host provisioning Cleanup of LVM thin pool volumes during cluster provisioning Wiping a device or partition before a bare metal cluster deployment Container Cloud web UI improvements Ceph: Support for Rook v1.12 Support for custom device classes Network policies for Rook Ceph daemons StackLight: Upgraded logging pipeline Support for custom labels during alert injection Documentation enhancements
2.25.4	Jan 10, 2024	Container Cloud 2.25.4 is the fourth patch release of the 2.25.x and MOSK 23.3.x release series that introduces the following updates: Patch Cluster release 17.0.4 for MOSK 23.3.4 Patch Cluster release 16.0.4 Security fixes for CVEs in images

2.28.3¶

Important

For MOSK clusters, Container Cloud 2.28.3 is the continuation for MOSK 24.2.x series using the patch Cluster release 17.2.7. For the update path of 24.1, 24.2, and 24.3 series, see MOSK documentation: Release Compatibility Matrix - Managed cluster update schema.

The management cluster of a MOSK 24.2.x cluster is automatically updated to the latest patch Cluster release 16.3.3.

The Container Cloud patch release 2.28.3, which is based on the 2.28.0 major release, provides the following updates:

Support for the patch Cluster release 16.3.3.
Support for the patch Cluster releases 16.2.7 and 17.2.7 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 24.2.5.
Bare metal: update of Ubuntu mirror from ubuntu-2024-10-28-012906 to ubuntu-2024-11-18-003900 along with update of minor kernel version from 5.15.0-124-generic to 5.15.0-125-generic.
Security fixes for CVEs in images.

For main deliverables of the parent Container Cloud release of 2.28.3, refer to 2.28.0.

Security notes¶

In total, since Container Cloud 2.28.2, 66 Common Vulnerabilities and Exposures (CVE) have been fixed in 2.28.3: 4 of critical and 62 of high severity.

The table below includes the total numbers of addressed unique and common CVEs in images by product component since Container Cloud 2.28.2. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	Critical	High	Total
Ceph	Unique	0	1	1
Ceph	Common	0	3	3
Kaas core	Unique	0	4	4
Kaas core	Common	0	7	7
StackLight	Unique	4	21	25
StackLight	Common	4	52	56

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 24.2.5: Security notes.

Addressed issues¶

The following issues have been addressed in the Container Cloud patch release 2.28.3 along with the patch Cluster releases 16.3.3, 16.2.7, and 17.2.7:

[47594] [StackLight] Fixed the issue with Patroni pods getting stuck in the CrashLoopBackOff state due to the patroni container being terminated with reason: OOMKilled.
[47929] [LCM] Fixed the issue with incorrect restrictive permissions set for registry certificate files in /etc/docker/certs.d, which were set to 644 instead of 444.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.28.3 including the Cluster releases 16.2.7, 16.3.3, and 17.2.7.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[47202] Inspection error on bare metal hosts after dnsmasq restart¶

Note

Moving forward, the workaround for this issue will be moved from Release Notes to MOSK Troubleshooting Guide: Inspection error on bare metal hosts after dnsmasq restart.

The issue can occur when:

The dnsmasq pod was moved to another node.
DHCP subnets were changed, including addition or removal. In this case, the dhcpd container of the dnsmasq pod is restarted.

Caution

If changing or adding of DHCP subnets is required to bootstrap new nodes, wait after changing or adding DHCP subnets until the dnsmasq pod becomes ready, then create BareMetalHost objects.

To verify whether the nodes are affected:

Verify whether the BareMetalHost objects contain the inspection error:

kubectl get bmh -n <managed-cluster-namespace-name>

Example of system response:

NAME            STATE         CONSUMER        ONLINE   ERROR              AGE
test-master-1   provisioned   test-master-1   true                        9d
test-master-2   provisioned   test-master-2   true                        9d
test-master-3   provisioned   test-master-3   true                        9d
test-worker-1   provisioned   test-worker-1   true                        9d
test-worker-2   provisioned   test-worker-2   true                        9d
test-worker-3   inspecting                    true     inspection error   19h

Verify whether the dnsmasq pod was in Ready state when the inspection of the affected baremetal hosts (test-worker-3 in the example above) was started:

kubectl -n kaas get pod <dnsmasq-pod-name> -oyaml

Example of system response:

...
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-10-10T15:37:34Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2024-10-11T07:38:54Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2024-10-11T07:38:54Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2024-10-10T15:37:34Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://6dbcf2fc4b36ce4c549c9191ab01f72d0236c51d42947675302675e4bfaf4cdf
    image: docker-dev-kaas-virtual.artifactory-eu.mcp.mirantis.net/bm/baremetal-dnsmasq:base-2-28-alpine-20240812132650
    imageID: docker-dev-kaas-virtual.artifactory-eu.mcp.mirantis.net/bm/baremetal-dnsmasq@sha256:3dad3e278add18e69b2608e462691c4823942641a0f0e25e6811e703e3c23b3b
    lastState:
      terminated:
        containerID: containerd://816fcf079cd544acd74e312065de5b5ed4dbf1dc6159fefffff4f644b5e45987
        exitCode: 0
        finishedAt: "2024-10-11T07:38:35Z"
        reason: Completed
        startedAt: "2024-10-10T15:37:45Z"
    name: dhcpd
    ready: true
    restartCount: 2
    started: true
    state:
      running:
        startedAt: "2024-10-11T07:38:37Z"
  ...

In the system response above, the dhcpd container was not ready between "2024-10-11T07:38:35Z" and "2024-10-11T07:38:54Z".

Verify the affected baremetal host. For example:

kubectl get bmh -n managed-ns test-worker-3 -oyaml

Example of system response:

...
status:
  errorCount: 15
  errorMessage: Introspection timeout
  errorType: inspection error
  ...
  operationHistory:
    deprovision:
      end: null
      start: null
    inspect:
      end: null
      start: "2024-10-11T07:38:19Z"
    provision:
      end: null
      start: null
    register:
      end: "2024-10-11T07:38:19Z"
      start: "2024-10-11T07:37:25Z"

Workaround

Reboot the node using the IPMI reset or cycle command.
If the node fails to boot, remove the failed BareMetalHost object and create it again:
1. Remove BareMetalHost object. For example:
```
kubectl delete bmh -n managed-ns test-worker-3
```
2. Verify that the BareMetalHost object is removed:
```
kubectl get bmh -n managed-ns test-worker-3
```
3. Create a BareMetalHost object from the template. For example:
```
kubectl create -f bmhc-test-worker-3.yaml
kubectl create -f bmh-test-worker-3.yaml
```

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

Ceph¶

[50566] Ceph upgrade is very slow during patch or major cluster update¶

Fixed in 2.29.3 (17.3.8, 16.3.8, and 16.4.3)

 Warning  Unhealthy  57s (x16 over 3m27s)  kubelet  Startup probe failed:
 ceph daemon health check failed with the following output:
> no valid command found; 10 closest matches:
> 0
> 1
> 2
> abort
> assert
> bluefs debug_inject_read_zeros
> bluefs files list
> bluefs stats
> bluestore bluefs device info [<alloc_size:int>]
> config diff
> admin_socket: invalid command

Workaround:

Complete the following steps during every patch or major cluster update of the Cluster releases 17.2.x, 17.3.x, and 17.4.x (until Ceph 18.2.5 becomes supported):

Plan extra time in the maintenance window for the patch cluster update.

Slow starts will still impact the update procedure, but after completing the following step, the recovery process noticeably shortens without affecting the overall cluster state and data responsiveness.
Select one of the following options:
- Before the cluster update, set the noout flag:
```
ceph osd set noout
```
  Once the Ceph OSDs image upgrade is done, unset the flag:
```
ceph osd unset noout
```
- Monitor the Ceph OSDs image upgrade. If the symptoms of slow start appear, set the noout flag as soon as possible. Once the Ceph OSDs image upgrade is done, unset the flag.

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

LCM¶

[39437] Failure to replace a master node on a Container Cloud cluster¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

During the replacement of a master node on a cluster of any type, the process may get stuck with Kubelet's NodeReady condition is Unknown in the machine status on the remaining master nodes.

As a workaround, log in on the affected node and run the following command:

docker restart ucp-kubelet

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

[30294] Replacement of a master node is stuck on the calico-node Pod start¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a master node on a cluster of any type, the calico-node Pod fails to start on a new node that has the same IP address as the node being replaced.

Workaround:

From a CLI with an MKE client bundle, create a shell alias to start calicoctl using the mirantis/ucp-dsinfo image:

Since MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/key.pem \
-e ETCD_CA_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/ca.pem \
-e ETCD_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/cert.pem \
-v /var/run/calico:/var/run/calico \
-v /var/lib/docker/volumes/ucp-kv-certs/_data:/var/lib/docker/volumes/ucp-kv-certs/_data:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl \
"

Before MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
-e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
-e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
-v /var/run/calico:/var/run/calico \
-v ucp-node-certs:/ucp-node-certs:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl --allow-version-mismatch \
"

In the above command, replace the following values with the corresponding settings of the affected cluster:

<etcdEndpoint> is the etcd endpoint defined in the Calico configuration file. For example, ETCD_ENDPOINTS=127.0.0.1:12378
<mkeVersion> is the MKE version installed on your cluster. For example, mirantis/ucp-dsinfo:3.5.7.

Verify the node list on the cluster:
```
kubectl get node
```
Compare this list with the node list in Calico to identify the old node:
```
calicoctl get node -o wide
```

Remove the old node from Calico:

calicoctl delete node kaas-node-<nodeID>

[5782] Manager machine fails to be deployed during node replacement¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a manager machine, the following problems may occur:

The system adds the node to Docker swarm but not to Kubernetes
The node Deployment gets stuck with failed RethinkDB health checks

Workaround:

Delete the failed node.
Wait for the MKE cluster to become healthy. To monitor the cluster status:
1. Log in to the MKE web UI as described in Connect to the Mirantis Kubernetes Engine web UI.
2. Monitor the cluster status as described in MKE Operations Guide: Monitor an MKE cluster with the MKE web UI.
Deploy a new node.

[5568] The calico-kube-controllers Pod fails to clean up resources¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During the unsafe or forced deletion of a manager machine running the calico-kube-controllers Pod in the kube-system namespace, the following issues occur:

The calico-kube-controllers Pod fails to clean up resources associated with the deleted node
The calico-node Pod may fail to start up on a newly created node if the machine is provisioned with the same IP address as the deleted machine had

As a workaround, before deletion of the node running the calico-kube-controllers Pod, cordon and drain the node:

kubectl cordon <nodeName>
kubectl drain <nodeName>

StackLight¶

[44193] OpenSearch reaches 85% disk usage watermark affecting the cluster state¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

Caution

The issue and the provided workaround apply only for clusters on which OpenSearch and Prometheus utilize the same storage pool.

To verify that the cluster is affected:

Verify the result of the following formula:
```
0.8 × OpenSearch_PVC_Size_GB + Prometheus_PVC_Size_GB > 0.85 × Total_Storage_Capacity_GB
```
In the formula, define the following values:
OpenSearch_PVC_Size_GB
Derived from .values.elasticsearch.persistentVolumeUsableStorageSizeGB, defaulting to .values.elasticsearch.persistentVolumeClaimSize if unspecified. To obtain the OpenSearch PVC size:
kubectl -n <namespaceName> get cluster <clusterName> -o yaml |\ yq '.spec.providerSpec.value.helmReleases[] | select(.name == "stacklight") | .values.elasticsearch.persistentVolumeClaimSize '
Example of system response:
10000Gi
Prometheus_PVC_Size_GB
Sourced from .values.prometheusServer.persistentVolumeClaimSize. To obtain the Prometheus PVC size:
kubectl -n <namespaceName> get cluster <clusterName> -o yaml |\ yq '.spec.providerSpec.value.helmReleases[] | select(.name == "stacklight") | .values.prometheusServer.persistentVolumeClaimSize '
Example of system response:
4000Gi
Total_Storage_Capacity_GB
Total capacity of the OpenSearch PVCs. For LVP, the capacity of the storage pool. To obtain the total capacity:
kubectl get pvc -n stacklight -l app=opensearch-master \ -o custom-columns=NAME:.metadata.name,CAPACITY:.status.capacity.storage
The system response contains multiple outputs, one per opensearch-master node. Select the capacity for the affected node.
Note

Convert the values to GB if they are set in different units.

If the formula result is positive, it is an early indication that the cluster is affected.
Verify whether the OpenSearchClusterStatusWarning or OpenSearchClusterStatusCritical alert is firing. And if so, verify the following:
1. Log in to the OpenSearch web UI.
2. In Management -> Dev Tools, run the following command:
```
GET _cluster/allocation/explain
```
  The following system response indicates that the corresponding node is affected:
```
"explanation": "the node is above the low watermark cluster setting \
[cluster.routing.allocation.disk.watermark.low=85%], using more disk space \
than the maximum allowed [85.0%], actual free: [xx.xxx%]"
```
  Note
  
  The system response may contain even higher watermark percent than 85.0%, depending on the case.

Workaround:

Warning

The workaround implies adjustement of the retention threshold for OpenSearch. And depending on the new threshold, some old logs will be deleted.

Adjust or set .values.elasticsearch.persistentVolumeUsableStorageSizeGB to a lower value for the affection check formula to be non-positive. For configuration details, see MOSK Operations Guide: StackLight configuration parameters - OpenSearch.

Mirantis also recommends reserving some space for other PVCs using storage from the pool. Use the following formula to calculate the required space:
```
persistentVolumeUsableStorageSizeGB =
0.84 × ((1 - Reserved_Percentage - Filesystem_Reserve) ×
Total_Storage_Capacity_GB - Prometheus_PVC_Size_GB) /
0.8
```
In the formula, define the following values:
Reserved_Percentage
A user-defined variable that specifies what percentage of the total storage capacity should not be used by OpenSearch or Prometheus. This is used to reserve space for other components. It should be expressed as a decimal. For example, for 5% of reservation, Reserved_Percentage is 0.05. Mirantis recommends using 0.05 as a starting point.

Filesystem_Reserve
Percentage to deduct for filesystems that may reserve some portion of the available storage, which is marked as occupied. For example, for EXT4, it is 5% by default, so the value must be 0.05.

Prometheus_PVC_Size_GB
Sourced from .values.prometheusServer.persistentVolumeClaimSize.

Total_Storage_Capacity_GB
Total capacity of the OpenSearch PVCs. For LVP, the capacity of the storage pool. To obtain the total capacity:
kubectl get pvc -n stacklight -l app=opensearch-master \ -o custom-columns=NAME:.metadata.name,CAPACITY:.status.capacity.storage
The system response contains multiple outputs, one per opensearch-master node. Select the capacity for the affected node.
Note

Convert the values to GB if they are set in different units.

Calculation of above formula provides a maximum safe storage to allocate for .values.elasticsearch.persistentVolumeUsableStorageSizeGB. Use this formula as a reference for setting .values.elasticsearch.persistentVolumeUsableStorageSizeGB on a cluster.
Wait up to 15-20 mins for OpenSearch to perform the cleaning.
Verify that the cluster is not affected anymore using the procedure above.

Container Cloud web UI¶

[50181] Failure to deploy a compact cluster¶

To work around the issue, manually add the required labels using CLI. Once done, the cluster deployment resumes.

[50168] Inability to use a new project right after creation¶

A newly created project does not display all available tabs in the Container Cloud web UI and contains different access denied errors during first five minutes after creation.

To work around the issue, refresh the browser in five minutes after the project creation.

Artifacts¶

This section lists the artifacts of components included in the Container Cloud patch release 2.28.3. For artifacts of the Cluster releases introduced in 2.28.3, see patch Cluster releases 17.2.7, 16.3.3, and 16.2.7.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries	ironic-python-agent.initramfs ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-antelope-jammy-debug-20241118155355
	ironic-python-agent.kernel ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-antelope-jammy-debug-20241118155355
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-167-e7a55fd.tgz
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.41.23.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.41.23.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.41.23.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.41.23.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.41.23.tgz
Docker images	ambassador ^Updated	mirantis.azurecr.io/core/external/nginx:1.41.23
	baremetal-dnsmasq	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-28-alpine-20241022121257
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-2-28-alpine-20241111132119
	bm-collective	mirantis.azurecr.io/bm/bm-collective:base-2-28-alpine-20241022120001
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.41.23
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:antelope-jammy-20241128095555
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:antelope-jammy-20241128095555
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20240913123302
	kaas-ipam	mirantis.azurecr.io/bm/kaas-ipam:base-2-28-alpine-20241022122006
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-34a4f54-20240910081335
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.17-jammy-20240927170336
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20241022120929

Core artifacts¶

Core artifacts¶
Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.41.23.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.41.23.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.41.23.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.41.23.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.41.23.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.41.23.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.41.23.tgz
	credentials-controller	https://binary.mirantis.com/core/helm/credentials-controller-1.41.23.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.41.23.tgz
	host-os-modules-controller	https://binary.mirantis.com/core/helm/host-os-modules-controller-1.41.23.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.41.23.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.41.23.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.41.23.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.41.23.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.41.23.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.41.23.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.41.23.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.41.23.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.41.23.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.41.23.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.41.23.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.41.23.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.41.23.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.41.23.tgz
	secret-controller	https://binary.mirantis.com/core/helm/secret-controller-1.41.23.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.41.23.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.41.23
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.41.23
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.41.23
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-8
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.41.23
	credentials-controller ^Updated	mirantis.azurecr.io/core/credentials-controller:1.41.23
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.41.23
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.41.23
	host-os-modules-controller ^Updated	mirantis.azurecr.io/core/host-os-modules-controller:1.41.23
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.41.23
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.41.23
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.41.23
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.41.23
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.41.23
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.41.23
	mcc-cache-warmup ^Updated	mirantis.azurecr.io/core/mcc-cache-warmup:1.41.23
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.41.23
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.41.23
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.41.23
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.41.23
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-14
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.41.23
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.41.23
	secret-controller ^Updated	mirantis.azurecr.io/core/secret-controller:1.41.23
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.41.23

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts ^Updated	iam	https://binary.mirantis.com/core/helm/iam-1.41.23.tgz
Docker images	kubectl	mirantis.azurecr.io/general/kubectl:20240926142019
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-ba8ada4-20240405150338
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.17-focal-20240909113408
	mcc-keycloak ^Updated	mirantis.azurecr.io/iam/mcc-keycloak:25.0.6-20241114073807

2.28.2¶

Important

For MOSK clusters, Container Cloud 2.28.2 is the continuation for MOSK 24.2.x series using the patch Cluster release 17.2.6. For the update path of 24.1, 24.2, and 24.3 series, see MOSK documentation: Release Compatibility Matrix - Managed cluster update schema.

The management cluster of a MOSK 24.2.x cluster is automatically updated to the latest patch Cluster release 16.3.2.

The Container Cloud patch release 2.28.2, which is based on the 2.28.0 major release, provides the following updates:

Support for the patch Cluster release 16.3.2.
Support for the patch Cluster releases 16.2.6 and 17.2.6 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 24.2.4.
Support for MKE 3.7.16.
Bare metal: update of Ubuntu mirror from ubuntu-2024-10-14-013948 to ubuntu-2024-10-28-012906 along with update of minor kernel version from 5.15.0-122-generic to 5.15.0-124-generic.
Security fixes for CVEs in images.

For main deliverables of the parent Container Cloud release of 2.28.2, refer to 2.28.0.

Security notes¶

In total, since Container Cloud 2.28.1, 15 Common Vulnerabilities and Exposures (CVE) of high severity have been fixed in 2.28.2.

The table below includes the total numbers of addressed unique and common CVEs in images by product component since Container Cloud 2.28.1. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	High	Total
Kaas core	Unique	5	5
Kaas core	Common	9	9
StackLight	Unique	1	1
StackLight	Common	6	6

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 24.2.4: Security notes.

Addressed issues¶

The following issues have been addressed in the Container Cloud patch release 2.28.2 along with the patch Cluster releases 16.3.2, 16.2.6, and 17.2.6.

[47741] [LCM] Fixed the issue with upgrade to MKE 3.7.15 getting stuck due to the leftover ucp-upgrade-check-images service that is part of MKE 3.7.12.
[47304] [StackLight] Fixed the issue with OpenSearch not storing kubelet logs due to the JSON-based format of ucp-kubelet logs.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.28.2 including the Cluster releases 16.2.6, 16.3.2, and 17.2.6.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[47202] Inspection error on bare metal hosts after dnsmasq restart¶

Note

Moving forward, the workaround for this issue will be moved from Release Notes to MOSK Troubleshooting Guide: Inspection error on bare metal hosts after dnsmasq restart.

The issue can occur when:

The dnsmasq pod was moved to another node.
DHCP subnets were changed, including addition or removal. In this case, the dhcpd container of the dnsmasq pod is restarted.

Caution

If changing or adding of DHCP subnets is required to bootstrap new nodes, wait after changing or adding DHCP subnets until the dnsmasq pod becomes ready, then create BareMetalHost objects.

To verify whether the nodes are affected:

Verify whether the BareMetalHost objects contain the inspection error:

kubectl get bmh -n <managed-cluster-namespace-name>

Example of system response:

NAME            STATE         CONSUMER        ONLINE   ERROR              AGE
test-master-1   provisioned   test-master-1   true                        9d
test-master-2   provisioned   test-master-2   true                        9d
test-master-3   provisioned   test-master-3   true                        9d
test-worker-1   provisioned   test-worker-1   true                        9d
test-worker-2   provisioned   test-worker-2   true                        9d
test-worker-3   inspecting                    true     inspection error   19h

Verify whether the dnsmasq pod was in Ready state when the inspection of the affected baremetal hosts (test-worker-3 in the example above) was started:

kubectl -n kaas get pod <dnsmasq-pod-name> -oyaml

Example of system response:

...
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-10-10T15:37:34Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2024-10-11T07:38:54Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2024-10-11T07:38:54Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2024-10-10T15:37:34Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://6dbcf2fc4b36ce4c549c9191ab01f72d0236c51d42947675302675e4bfaf4cdf
    image: docker-dev-kaas-virtual.artifactory-eu.mcp.mirantis.net/bm/baremetal-dnsmasq:base-2-28-alpine-20240812132650
    imageID: docker-dev-kaas-virtual.artifactory-eu.mcp.mirantis.net/bm/baremetal-dnsmasq@sha256:3dad3e278add18e69b2608e462691c4823942641a0f0e25e6811e703e3c23b3b
    lastState:
      terminated:
        containerID: containerd://816fcf079cd544acd74e312065de5b5ed4dbf1dc6159fefffff4f644b5e45987
        exitCode: 0
        finishedAt: "2024-10-11T07:38:35Z"
        reason: Completed
        startedAt: "2024-10-10T15:37:45Z"
    name: dhcpd
    ready: true
    restartCount: 2
    started: true
    state:
      running:
        startedAt: "2024-10-11T07:38:37Z"
  ...

In the system response above, the dhcpd container was not ready between "2024-10-11T07:38:35Z" and "2024-10-11T07:38:54Z".

Verify the affected baremetal host. For example:

kubectl get bmh -n managed-ns test-worker-3 -oyaml

Example of system response:

...
status:
  errorCount: 15
  errorMessage: Introspection timeout
  errorType: inspection error
  ...
  operationHistory:
    deprovision:
      end: null
      start: null
    inspect:
      end: null
      start: "2024-10-11T07:38:19Z"
    provision:
      end: null
      start: null
    register:
      end: "2024-10-11T07:38:19Z"
      start: "2024-10-11T07:37:25Z"

Workaround

Reboot the node using the IPMI reset or cycle command.
If the node fails to boot, remove the failed BareMetalHost object and create it again:
1. Remove BareMetalHost object. For example:
```
kubectl delete bmh -n managed-ns test-worker-3
```
2. Verify that the BareMetalHost object is removed:
```
kubectl get bmh -n managed-ns test-worker-3
```
3. Create a BareMetalHost object from the template. For example:
```
kubectl create -f bmhc-test-worker-3.yaml
kubectl create -f bmh-test-worker-3.yaml
```

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

Ceph¶

[50566] Ceph upgrade is very slow during patch or major cluster update¶

Fixed in 2.29.3 (17.3.8, 16.3.8, and 16.4.3)

 Warning  Unhealthy  57s (x16 over 3m27s)  kubelet  Startup probe failed:
 ceph daemon health check failed with the following output:
> no valid command found; 10 closest matches:
> 0
> 1
> 2
> abort
> assert
> bluefs debug_inject_read_zeros
> bluefs files list
> bluefs stats
> bluestore bluefs device info [<alloc_size:int>]
> config diff
> admin_socket: invalid command

Workaround:

Complete the following steps during every patch or major cluster update of the Cluster releases 17.2.x, 17.3.x, and 17.4.x (until Ceph 18.2.5 becomes supported):

Plan extra time in the maintenance window for the patch cluster update.

Slow starts will still impact the update procedure, but after completing the following step, the recovery process noticeably shortens without affecting the overall cluster state and data responsiveness.
Select one of the following options:
- Before the cluster update, set the noout flag:
```
ceph osd set noout
```
  Once the Ceph OSDs image upgrade is done, unset the flag:
```
ceph osd unset noout
```
- Monitor the Ceph OSDs image upgrade. If the symptoms of slow start appear, set the noout flag as soon as possible. Once the Ceph OSDs image upgrade is done, unset the flag.

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

LCM¶

[39437] Failure to replace a master node on a Container Cloud cluster¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

During the replacement of a master node on a cluster of any type, the process may get stuck with Kubelet's NodeReady condition is Unknown in the machine status on the remaining master nodes.

As a workaround, log in on the affected node and run the following command:

docker restart ucp-kubelet

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

[30294] Replacement of a master node is stuck on the calico-node Pod start¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a master node on a cluster of any type, the calico-node Pod fails to start on a new node that has the same IP address as the node being replaced.

Workaround:

From a CLI with an MKE client bundle, create a shell alias to start calicoctl using the mirantis/ucp-dsinfo image:

Since MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/key.pem \
-e ETCD_CA_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/ca.pem \
-e ETCD_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/cert.pem \
-v /var/run/calico:/var/run/calico \
-v /var/lib/docker/volumes/ucp-kv-certs/_data:/var/lib/docker/volumes/ucp-kv-certs/_data:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl \
"

Before MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
-e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
-e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
-v /var/run/calico:/var/run/calico \
-v ucp-node-certs:/ucp-node-certs:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl --allow-version-mismatch \
"

In the above command, replace the following values with the corresponding settings of the affected cluster:

<etcdEndpoint> is the etcd endpoint defined in the Calico configuration file. For example, ETCD_ENDPOINTS=127.0.0.1:12378
<mkeVersion> is the MKE version installed on your cluster. For example, mirantis/ucp-dsinfo:3.5.7.

Verify the node list on the cluster:
```
kubectl get node
```
Compare this list with the node list in Calico to identify the old node:
```
calicoctl get node -o wide
```

Remove the old node from Calico:

calicoctl delete node kaas-node-<nodeID>

[5782] Manager machine fails to be deployed during node replacement¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a manager machine, the following problems may occur:

The system adds the node to Docker swarm but not to Kubernetes
The node Deployment gets stuck with failed RethinkDB health checks

Workaround:

Delete the failed node.
Wait for the MKE cluster to become healthy. To monitor the cluster status:
1. Log in to the MKE web UI as described in Connect to the Mirantis Kubernetes Engine web UI.
2. Monitor the cluster status as described in MKE Operations Guide: Monitor an MKE cluster with the MKE web UI.
Deploy a new node.

[5568] The calico-kube-controllers Pod fails to clean up resources¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During the unsafe or forced deletion of a manager machine running the calico-kube-controllers Pod in the kube-system namespace, the following issues occur:

The calico-kube-controllers Pod fails to clean up resources associated with the deleted node
The calico-node Pod may fail to start up on a newly created node if the machine is provisioned with the same IP address as the deleted machine had

As a workaround, before deletion of the node running the calico-kube-controllers Pod, cordon and drain the node:

kubectl cordon <nodeName>
kubectl drain <nodeName>

StackLight¶

[47594] Patroni pods may get stuck in the CrashLoopBackOff state¶

Fixed in 2.28.3 (17.2.7, 16.2.7, and 16.3.3)

The Patroni pods may get stuck in the CrashLoopBackOff state due to the patroni container being terminated with reason: OOMKilled that you can see in the pod status. For example:

kubectl get pod/patroni-13-0 -n stacklight -o yaml

...
  - containerID: docker://<ID>`
    image: mirantis.azurecr.io/stacklight/spilo:13-2.1p9-20240828023010
    imageID: docker-pullable://mirantis.azurecr.io/stacklight/spilo@sha256:<ID>
    lastState:
      terminated:
        containerID: docker://<ID>
        exitCode: 137
        finishedAt: "2024-10-17T14:26:25Z"
        reason: OOMKilled
        startedAt: "2024-10-17T14:23:25Z"
    name: patroni
...

As a workaround, increase the memory limit for PostgreSQL to 20Gi in the Cluster object:

spec:
  providerSpec:
    value:
      helmReleases:
      - name: stacklight
        values:
          resources:
            postgresql:
              limits:
                memory: "20Gi"

For a detailed procedure of StackLight configuration, see MOSK Operations Guide: Configure StackLight. For description of the resources option, see MOSK Operations Guide: StackLight configuration parameters - Resource limits.

[44193] OpenSearch reaches 85% disk usage watermark affecting the cluster state¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

Caution

The issue and the provided workaround apply only for clusters on which OpenSearch and Prometheus utilize the same storage pool.

To verify that the cluster is affected:

Verify the result of the following formula:
```
0.8 × OpenSearch_PVC_Size_GB + Prometheus_PVC_Size_GB > 0.85 × Total_Storage_Capacity_GB
```
In the formula, define the following values:
OpenSearch_PVC_Size_GB
Derived from .values.elasticsearch.persistentVolumeUsableStorageSizeGB, defaulting to .values.elasticsearch.persistentVolumeClaimSize if unspecified. To obtain the OpenSearch PVC size:
kubectl -n <namespaceName> get cluster <clusterName> -o yaml |\ yq '.spec.providerSpec.value.helmReleases[] | select(.name == "stacklight") | .values.elasticsearch.persistentVolumeClaimSize '
Example of system response:
10000Gi
Prometheus_PVC_Size_GB
Sourced from .values.prometheusServer.persistentVolumeClaimSize. To obtain the Prometheus PVC size:
kubectl -n <namespaceName> get cluster <clusterName> -o yaml |\ yq '.spec.providerSpec.value.helmReleases[] | select(.name == "stacklight") | .values.prometheusServer.persistentVolumeClaimSize '
Example of system response:
4000Gi
Total_Storage_Capacity_GB
Total capacity of the OpenSearch PVCs. For LVP, the capacity of the storage pool. To obtain the total capacity:
kubectl get pvc -n stacklight -l app=opensearch-master \ -o custom-columns=NAME:.metadata.name,CAPACITY:.status.capacity.storage
The system response contains multiple outputs, one per opensearch-master node. Select the capacity for the affected node.
Note

Convert the values to GB if they are set in different units.

If the formula result is positive, it is an early indication that the cluster is affected.
Verify whether the OpenSearchClusterStatusWarning or OpenSearchClusterStatusCritical alert is firing. And if so, verify the following:
1. Log in to the OpenSearch web UI.
2. In Management -> Dev Tools, run the following command:
```
GET _cluster/allocation/explain
```
  The following system response indicates that the corresponding node is affected:
```
"explanation": "the node is above the low watermark cluster setting \
[cluster.routing.allocation.disk.watermark.low=85%], using more disk space \
than the maximum allowed [85.0%], actual free: [xx.xxx%]"
```
  Note
  
  The system response may contain even higher watermark percent than 85.0%, depending on the case.

Workaround:

Warning

The workaround implies adjustement of the retention threshold for OpenSearch. And depending on the new threshold, some old logs will be deleted.

Adjust or set .values.elasticsearch.persistentVolumeUsableStorageSizeGB to a lower value for the affection check formula to be non-positive. For configuration details, see MOSK Operations Guide: StackLight configuration parameters - OpenSearch.

Mirantis also recommends reserving some space for other PVCs using storage from the pool. Use the following formula to calculate the required space:
```
persistentVolumeUsableStorageSizeGB =
0.84 × ((1 - Reserved_Percentage - Filesystem_Reserve) ×
Total_Storage_Capacity_GB - Prometheus_PVC_Size_GB) /
0.8
```
In the formula, define the following values:
Reserved_Percentage
A user-defined variable that specifies what percentage of the total storage capacity should not be used by OpenSearch or Prometheus. This is used to reserve space for other components. It should be expressed as a decimal. For example, for 5% of reservation, Reserved_Percentage is 0.05. Mirantis recommends using 0.05 as a starting point.

Filesystem_Reserve
Percentage to deduct for filesystems that may reserve some portion of the available storage, which is marked as occupied. For example, for EXT4, it is 5% by default, so the value must be 0.05.

Prometheus_PVC_Size_GB
Sourced from .values.prometheusServer.persistentVolumeClaimSize.

Total_Storage_Capacity_GB
Total capacity of the OpenSearch PVCs. For LVP, the capacity of the storage pool. To obtain the total capacity:
kubectl get pvc -n stacklight -l app=opensearch-master \ -o custom-columns=NAME:.metadata.name,CAPACITY:.status.capacity.storage
The system response contains multiple outputs, one per opensearch-master node. Select the capacity for the affected node.
Note

Convert the values to GB if they are set in different units.

Calculation of above formula provides a maximum safe storage to allocate for .values.elasticsearch.persistentVolumeUsableStorageSizeGB. Use this formula as a reference for setting .values.elasticsearch.persistentVolumeUsableStorageSizeGB on a cluster.
Wait up to 15-20 mins for OpenSearch to perform the cleaning.
Verify that the cluster is not affected anymore using the procedure above.

Container Cloud web UI¶

[50181] Failure to deploy a compact cluster¶

To work around the issue, manually add the required labels using CLI. Once done, the cluster deployment resumes.

[50168] Inability to use a new project right after creation¶

A newly created project does not display all available tabs in the Container Cloud web UI and contains different access denied errors during first five minutes after creation.

To work around the issue, refresh the browser in five minutes after the project creation.

Artifacts¶

This section lists the artifacts of components included in the Container Cloud patch release 2.28.2. For artifacts of the Cluster releases introduced in 2.28.2, see patch Cluster releases 17.2.6, 16.3.2, and 16.2.6.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries ^Updated	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-antelope-jammy-debug-20241028161924
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-antelope-jammy-debug-20241028161924
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.41.22.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.41.22.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.41.22.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.41.22.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.41.22.tgz
Docker images	ambasador ^Updated	mirantis.azurecr.io/core/external/nginx:1.41.22
	baremetal-dnsmasq	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-28-alpine-20241022121257
	baremetal-operator	mirantis.azurecr.io/bm/baremetal-operator:base-2-28-alpine-20241022120949
	bm-collective	mirantis.azurecr.io/bm/bm-collective:base-2-28-alpine-20241022120001
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.41.22
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:antelope-jammy-20241023091304
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:antelope-jammy-20241023091304
	ironic-prometheus-exporter ^Updated	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20240913123302
	kaas-ipam	mirantis.azurecr.io/bm/kaas-ipam:base-2-28-alpine-20241022122006
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-34a4f54-20240910081335
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.17-jammy-20240927170336
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20241022120929

Core artifacts¶

Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.41.22.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.41.22.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.41.22.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.41.22.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.41.22.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.41.22.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.41.22.tgz
	credentials-controller	https://binary.mirantis.com/core/helm/credentials-controller-1.41.22.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.41.22.tgz
	host-os-modules-controller	https://binary.mirantis.com/core/helm/host-os-modules-controller-1.41.22.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.41.22.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.41.22.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.41.22.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.41.22.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.41.22.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.41.22.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.41.22.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.41.22.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.41.22.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.41.22.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.41.22.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.41.22.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.41.22.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.41.22.tgz
	secret-controller	https://binary.mirantis.com/core/helm/secret-controller-1.41.22.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.41.22.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.41.22
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.41.22
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.41.22
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-8
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.41.22
	credentials-controller ^Updated	mirantis.azurecr.io/core/credentials-controller:1.41.22
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.41.22
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.41.22
	host-os-modules-controller ^Updated	mirantis.azurecr.io/core/host-os-modules-controller:1.41.22
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.41.22
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.41.22
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.41.22
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.41.22
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.41.22
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.41.22
	mcc-cache-warmup ^Updated	mirantis.azurecr.io/core/mcc-cache-warmup:1.41.22
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.41.22
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.41.22
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.41.22
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.41.22
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-14
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.41.22
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.41.22
	secret-controller ^Updated	mirantis.azurecr.io/core/secret-controller:1.41.22
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.41.22

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts ^Updated	iam	https://binary.mirantis.com/core/helm/iam-1.41.22.tgz
Docker images	kubectl	mirantis.azurecr.io/general/kubectl:20240926142019
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-ba8ada4-20240405150338
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.17-focal-20240909113408
	mcc-keycloak	mirantis.azurecr.io/iam/mcc-keycloak:25.0.6-20240926140203

2.28.1¶

Important

For MOSK clusters, Container Cloud 2.28.1 is the continuation for MOSK 24.2.x series using the patch Cluster release 17.2.5. For the update path of 24.1, 24.2, and 24.3 series, see MOSK documentation: Release Compatibility Matrix - Managed cluster update schema.

The management cluster of a MOSK 24.2.x cluster is automatically updated to the latest patch Cluster release 16.3.1.

The Container Cloud patch release 2.28.1, which is based on the 2.28.0 major release, provides the following updates:

Support for the patch Cluster release 16.3.1.
Support for the patch Cluster releases 16.2.5 and 17.2.5 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 24.2.3.
Support for MKE 3.7.15.
Bare metal: update of Ubuntu mirror from 2024-09-11-014225 to ubuntu-2024-10-14-013948 along with update of minor kernel version from 5.15.0-119-generic to 5.15.0-122-generic.
Security fixes for CVEs in images.

For main deliverables of the parent Container Cloud release of 2.28.1, refer to 2.28.0.

Security notes¶

In total, since Container Cloud 2.28.0, 400 Common Vulnerabilities and Exposures (CVE) have been fixed in 2.28.1: 46 of critical and 354 of high severity.

The table below includes the total numbers of addressed unique and common CVEs in images by product component since Container Cloud 2.28.0. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	Critical	High	Total
Ceph	Unique	0	1	1
Ceph	Common	0	4	4
Kaas core	Unique	1	14	15
Kaas core	Common	1	118	119
StackLight	Unique	8	40	48
StackLight	Common	45	232	277

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 24.2.3: Security notes.

Addressed issues¶

The following issues have been addressed in the Container Cloud patch release 2.28.1 along with the patch Cluster releases 16.3.1, 16.2.5, and 17.2.5.

[46808] [LCM] Fixed the issue with old kernel metapackages remaining on the cluster after kernel upgrade.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.28.1 including the Cluster releases 16.2.5, 16.3.1, and 17.2.5.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[47202] Inspection error on bare metal hosts after dnsmasq restart¶

Note

Moving forward, the workaround for this issue will be moved from Release Notes to MOSK Troubleshooting Guide: Inspection error on bare metal hosts after dnsmasq restart.

The issue can occur when:

The dnsmasq pod was moved to another node.
DHCP subnets were changed, including addition or removal. In this case, the dhcpd container of the dnsmasq pod is restarted.

Caution

If changing or adding of DHCP subnets is required to bootstrap new nodes, wait after changing or adding DHCP subnets until the dnsmasq pod becomes ready, then create BareMetalHost objects.

To verify whether the nodes are affected:

Verify whether the BareMetalHost objects contain the inspection error:

kubectl get bmh -n <managed-cluster-namespace-name>

Example of system response:

NAME            STATE         CONSUMER        ONLINE   ERROR              AGE
test-master-1   provisioned   test-master-1   true                        9d
test-master-2   provisioned   test-master-2   true                        9d
test-master-3   provisioned   test-master-3   true                        9d
test-worker-1   provisioned   test-worker-1   true                        9d
test-worker-2   provisioned   test-worker-2   true                        9d
test-worker-3   inspecting                    true     inspection error   19h

Verify whether the dnsmasq pod was in Ready state when the inspection of the affected baremetal hosts (test-worker-3 in the example above) was started:

kubectl -n kaas get pod <dnsmasq-pod-name> -oyaml

Example of system response:

...
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-10-10T15:37:34Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2024-10-11T07:38:54Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2024-10-11T07:38:54Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2024-10-10T15:37:34Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://6dbcf2fc4b36ce4c549c9191ab01f72d0236c51d42947675302675e4bfaf4cdf
    image: docker-dev-kaas-virtual.artifactory-eu.mcp.mirantis.net/bm/baremetal-dnsmasq:base-2-28-alpine-20240812132650
    imageID: docker-dev-kaas-virtual.artifactory-eu.mcp.mirantis.net/bm/baremetal-dnsmasq@sha256:3dad3e278add18e69b2608e462691c4823942641a0f0e25e6811e703e3c23b3b
    lastState:
      terminated:
        containerID: containerd://816fcf079cd544acd74e312065de5b5ed4dbf1dc6159fefffff4f644b5e45987
        exitCode: 0
        finishedAt: "2024-10-11T07:38:35Z"
        reason: Completed
        startedAt: "2024-10-10T15:37:45Z"
    name: dhcpd
    ready: true
    restartCount: 2
    started: true
    state:
      running:
        startedAt: "2024-10-11T07:38:37Z"
  ...

In the system response above, the dhcpd container was not ready between "2024-10-11T07:38:35Z" and "2024-10-11T07:38:54Z".

Verify the affected baremetal host. For example:

kubectl get bmh -n managed-ns test-worker-3 -oyaml

Example of system response:

...
status:
  errorCount: 15
  errorMessage: Introspection timeout
  errorType: inspection error
  ...
  operationHistory:
    deprovision:
      end: null
      start: null
    inspect:
      end: null
      start: "2024-10-11T07:38:19Z"
    provision:
      end: null
      start: null
    register:
      end: "2024-10-11T07:38:19Z"
      start: "2024-10-11T07:37:25Z"

Workaround

Reboot the node using the IPMI reset or cycle command.
If the node fails to boot, remove the failed BareMetalHost object and create it again:
1. Remove BareMetalHost object. For example:
```
kubectl delete bmh -n managed-ns test-worker-3
```
2. Verify that the BareMetalHost object is removed:
```
kubectl get bmh -n managed-ns test-worker-3
```
3. Create a BareMetalHost object from the template. For example:
```
kubectl create -f bmhc-test-worker-3.yaml
kubectl create -f bmh-test-worker-3.yaml
```

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

Ceph¶

[50566] Ceph upgrade is very slow during patch or major cluster update¶

Fixed in 2.29.3 (17.3.8, 16.3.8, and 16.4.3)

 Warning  Unhealthy  57s (x16 over 3m27s)  kubelet  Startup probe failed:
 ceph daemon health check failed with the following output:
> no valid command found; 10 closest matches:
> 0
> 1
> 2
> abort
> assert
> bluefs debug_inject_read_zeros
> bluefs files list
> bluefs stats
> bluestore bluefs device info [<alloc_size:int>]
> config diff
> admin_socket: invalid command

Workaround:

Complete the following steps during every patch or major cluster update of the Cluster releases 17.2.x, 17.3.x, and 17.4.x (until Ceph 18.2.5 becomes supported):

Plan extra time in the maintenance window for the patch cluster update.

Slow starts will still impact the update procedure, but after completing the following step, the recovery process noticeably shortens without affecting the overall cluster state and data responsiveness.
Select one of the following options:
- Before the cluster update, set the noout flag:
```
ceph osd set noout
```
  Once the Ceph OSDs image upgrade is done, unset the flag:
```
ceph osd unset noout
```
- Monitor the Ceph OSDs image upgrade. If the symptoms of slow start appear, set the noout flag as soon as possible. Once the Ceph OSDs image upgrade is done, unset the flag.

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

LCM¶

[47741] Upgrade to MKE 3.7.15 is blocked by ucp-upgrade-check-images¶

Fixed in 2.28.2 (17.2.6, 16.2.6, and 16.3.2)

Upgrade from MKE 3.7.12 to 3.7.15 may get stuck due to the leftover ucp-upgrade-check-images service that is part of MKE 3.7.12.

As a workaround, on any master node, remove the leftover service using the docker service rm ucp-upgrade-check-images command.

[39437] Failure to replace a master node on a Container Cloud cluster¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

During the replacement of a master node on a cluster of any type, the process may get stuck with Kubelet's NodeReady condition is Unknown in the machine status on the remaining master nodes.

As a workaround, log in on the affected node and run the following command:

docker restart ucp-kubelet

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

[30294] Replacement of a master node is stuck on the calico-node Pod start¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a master node on a cluster of any type, the calico-node Pod fails to start on a new node that has the same IP address as the node being replaced.

Workaround:

From a CLI with an MKE client bundle, create a shell alias to start calicoctl using the mirantis/ucp-dsinfo image:

Since MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/key.pem \
-e ETCD_CA_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/ca.pem \
-e ETCD_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/cert.pem \
-v /var/run/calico:/var/run/calico \
-v /var/lib/docker/volumes/ucp-kv-certs/_data:/var/lib/docker/volumes/ucp-kv-certs/_data:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl \
"

Before MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
-e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
-e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
-v /var/run/calico:/var/run/calico \
-v ucp-node-certs:/ucp-node-certs:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl --allow-version-mismatch \
"

In the above command, replace the following values with the corresponding settings of the affected cluster:

<etcdEndpoint> is the etcd endpoint defined in the Calico configuration file. For example, ETCD_ENDPOINTS=127.0.0.1:12378
<mkeVersion> is the MKE version installed on your cluster. For example, mirantis/ucp-dsinfo:3.5.7.

Verify the node list on the cluster:
```
kubectl get node
```
Compare this list with the node list in Calico to identify the old node:
```
calicoctl get node -o wide
```

Remove the old node from Calico:

calicoctl delete node kaas-node-<nodeID>

[5782] Manager machine fails to be deployed during node replacement¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a manager machine, the following problems may occur:

The system adds the node to Docker swarm but not to Kubernetes
The node Deployment gets stuck with failed RethinkDB health checks

Workaround:

Delete the failed node.
Wait for the MKE cluster to become healthy. To monitor the cluster status:
1. Log in to the MKE web UI as described in Connect to the Mirantis Kubernetes Engine web UI.
2. Monitor the cluster status as described in MKE Operations Guide: Monitor an MKE cluster with the MKE web UI.
Deploy a new node.

[5568] The calico-kube-controllers Pod fails to clean up resources¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During the unsafe or forced deletion of a manager machine running the calico-kube-controllers Pod in the kube-system namespace, the following issues occur:

The calico-kube-controllers Pod fails to clean up resources associated with the deleted node
The calico-node Pod may fail to start up on a newly created node if the machine is provisioned with the same IP address as the deleted machine had

As a workaround, before deletion of the node running the calico-kube-controllers Pod, cordon and drain the node:

kubectl cordon <nodeName>
kubectl drain <nodeName>

StackLight¶

[47594] Patroni pods may get stuck in the CrashLoopBackOff state¶

Fixed in 2.28.3 (17.2.7, 16.2.7, and 16.3.3)

The Patroni pods may get stuck in the CrashLoopBackOff state due to the patroni container being terminated with reason: OOMKilled that you can see in the pod status. For example:

kubectl get pod/patroni-13-0 -n stacklight -o yaml

...
  - containerID: docker://<ID>`
    image: mirantis.azurecr.io/stacklight/spilo:13-2.1p9-20240828023010
    imageID: docker-pullable://mirantis.azurecr.io/stacklight/spilo@sha256:<ID>
    lastState:
      terminated:
        containerID: docker://<ID>
        exitCode: 137
        finishedAt: "2024-10-17T14:26:25Z"
        reason: OOMKilled
        startedAt: "2024-10-17T14:23:25Z"
    name: patroni
...

As a workaround, increase the memory limit for PostgreSQL to 20Gi in the Cluster object:

spec:
  providerSpec:
    value:
      helmReleases:
      - name: stacklight
        values:
          resources:
            postgresql:
              limits:
                memory: "20Gi"

[47304] OpenSearch does not store kubelet logs¶

Fixed in 2.28.2 (17.2.6, 16.2.6, and 16.3.2)

Due to the JSON-based format of ucp-kubelet logs, OpenSearch does not store kubelet logs. Mirantis is working on the issue and will deliver the resolution in one of the nearest patch releases.

[44193] OpenSearch reaches 85% disk usage watermark affecting the cluster state¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

Caution

The issue and the provided workaround apply only for clusters on which OpenSearch and Prometheus utilize the same storage pool.

To verify that the cluster is affected:

Verify the result of the following formula:
```
0.8 × OpenSearch_PVC_Size_GB + Prometheus_PVC_Size_GB > 0.85 × Total_Storage_Capacity_GB
```
In the formula, define the following values:
OpenSearch_PVC_Size_GB
Derived from .values.elasticsearch.persistentVolumeUsableStorageSizeGB, defaulting to .values.elasticsearch.persistentVolumeClaimSize if unspecified. To obtain the OpenSearch PVC size:
kubectl -n <namespaceName> get cluster <clusterName> -o yaml |\ yq '.spec.providerSpec.value.helmReleases[] | select(.name == "stacklight") | .values.elasticsearch.persistentVolumeClaimSize '
Example of system response:
10000Gi
Prometheus_PVC_Size_GB
Sourced from .values.prometheusServer.persistentVolumeClaimSize. To obtain the Prometheus PVC size:
kubectl -n <namespaceName> get cluster <clusterName> -o yaml |\ yq '.spec.providerSpec.value.helmReleases[] | select(.name == "stacklight") | .values.prometheusServer.persistentVolumeClaimSize '
Example of system response:
4000Gi
Total_Storage_Capacity_GB
Total capacity of the OpenSearch PVCs. For LVP, the capacity of the storage pool. To obtain the total capacity:
kubectl get pvc -n stacklight -l app=opensearch-master \ -o custom-columns=NAME:.metadata.name,CAPACITY:.status.capacity.storage
The system response contains multiple outputs, one per opensearch-master node. Select the capacity for the affected node.
Note

Convert the values to GB if they are set in different units.

If the formula result is positive, it is an early indication that the cluster is affected.
Verify whether the OpenSearchClusterStatusWarning or OpenSearchClusterStatusCritical alert is firing. And if so, verify the following:
1. Log in to the OpenSearch web UI.
2. In Management -> Dev Tools, run the following command:
```
GET _cluster/allocation/explain
```
  The following system response indicates that the corresponding node is affected:
```
"explanation": "the node is above the low watermark cluster setting \
[cluster.routing.allocation.disk.watermark.low=85%], using more disk space \
than the maximum allowed [85.0%], actual free: [xx.xxx%]"
```
  Note
  
  The system response may contain even higher watermark percent than 85.0%, depending on the case.

Workaround:

Warning

The workaround implies adjustement of the retention threshold for OpenSearch. And depending on the new threshold, some old logs will be deleted.

Adjust or set .values.elasticsearch.persistentVolumeUsableStorageSizeGB to a lower value for the affection check formula to be non-positive. For configuration details, see MOSK Operations Guide: StackLight configuration parameters - OpenSearch.

Mirantis also recommends reserving some space for other PVCs using storage from the pool. Use the following formula to calculate the required space:
```
persistentVolumeUsableStorageSizeGB =
0.84 × ((1 - Reserved_Percentage - Filesystem_Reserve) ×
Total_Storage_Capacity_GB - Prometheus_PVC_Size_GB) /
0.8
```
In the formula, define the following values:
Reserved_Percentage
A user-defined variable that specifies what percentage of the total storage capacity should not be used by OpenSearch or Prometheus. This is used to reserve space for other components. It should be expressed as a decimal. For example, for 5% of reservation, Reserved_Percentage is 0.05. Mirantis recommends using 0.05 as a starting point.

Filesystem_Reserve
Percentage to deduct for filesystems that may reserve some portion of the available storage, which is marked as occupied. For example, for EXT4, it is 5% by default, so the value must be 0.05.

Prometheus_PVC_Size_GB
Sourced from .values.prometheusServer.persistentVolumeClaimSize.

Total_Storage_Capacity_GB
Total capacity of the OpenSearch PVCs. For LVP, the capacity of the storage pool. To obtain the total capacity:
kubectl get pvc -n stacklight -l app=opensearch-master \ -o custom-columns=NAME:.metadata.name,CAPACITY:.status.capacity.storage
The system response contains multiple outputs, one per opensearch-master node. Select the capacity for the affected node.
Note

Convert the values to GB if they are set in different units.

Calculation of above formula provides a maximum safe storage to allocate for .values.elasticsearch.persistentVolumeUsableStorageSizeGB. Use this formula as a reference for setting .values.elasticsearch.persistentVolumeUsableStorageSizeGB on a cluster.
Wait up to 15-20 mins for OpenSearch to perform the cleaning.
Verify that the cluster is not affected anymore using the procedure above.

Container Cloud web UI¶

[50181] Failure to deploy a compact cluster¶

To work around the issue, manually add the required labels using CLI. Once done, the cluster deployment resumes.

[50168] Inability to use a new project right after creation¶

A newly created project does not display all available tabs in the Container Cloud web UI and contains different access denied errors during first five minutes after creation.

To work around the issue, refresh the browser in five minutes after the project creation.

Artifacts¶

This section lists the artifacts of components included in the Container Cloud patch release 2.28.1. For artifacts of the Cluster releases introduced in 2.28.1, see patch Cluster releases 17.2.5, 16.3.1, and 16.2.5.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries ^Updated	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20241014163420
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20241014163420
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.41.18.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.41.18.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.41.18.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.41.18.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.41.18.tgz
Docker images	ambasador ^Updated	mirantis.azurecr.io/core/external/nginx:1.41.18
	baremetal-dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-28-alpine-20241022121257
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-2-28-alpine-20241022120949
	bm-collective ^Updated	mirantis.azurecr.io/bm/bm-collective:base-2-28-alpine-20241022120001
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.41.18
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:antelope-jammy-20240927160001
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:antelope-jammy-20240927160001
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20240819102310
	kaas-ipam ^Updated	mirantis.azurecr.io/bm/kaas-ipam:base-2-28-alpine-20241022122006
	kubernetes-entrypoint ^Updated	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-34a4f54-20240910081335
	mariadb ^Updated	mirantis.azurecr.io/general/mariadb:10.6.17-jammy-20240927170336
	syslog-ng ^Updated	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20241022120929

Core artifacts¶

Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.41.18.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.41.18.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.41.18.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.41.18.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.41.18.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.41.18.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.41.18.tgz
	credentials-controller	https://binary.mirantis.com/core/helm/credentials-controller-1.41.18.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.41.18.tgz
	host-os-modules-controller	https://binary.mirantis.com/core/helm/host-os-modules-controller-1.41.18.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.41.18.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.41.18.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.41.18.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.41.18.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.41.18.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.41.18.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.41.18.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.41.18.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.41.18.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.41.18.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.41.18.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.41.18.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.41.18.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.41.18.tgz
	secret-controller	https://binary.mirantis.com/core/helm/secret-controller-1.41.18.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.41.18.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.41.18
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.41.18
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.41.18
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-8
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.41.18
	credentials-controller ^Updated	mirantis.azurecr.io/core/credentials-controller:1.41.18
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.41.18
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.41.18
	host-os-modules-controller ^Updated	mirantis.azurecr.io/core/host-os-modules-controller:1.41.18
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.41.18
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.41.18
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.41.18
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.41.18
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.41.18
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.41.18
	mcc-cache-warmup ^Updated	mirantis.azurecr.io/core/mcc-cache-warmup:1.41.18
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.41.18
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.41.18
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.41.18
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.41.18
	registry ^Updated	mirantis.azurecr.io/lcm/registry:v2.8.1-14
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.41.18
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.41.18
	secret-controller ^Updated	mirantis.azurecr.io/core/secret-controller:1.41.18
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.41.18

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts ^Updated	iam	https://binary.mirantis.com/core/helm/iam-1.41.18.tgz
Docker images	kubectl	mirantis.azurecr.io/general/kubectl:20240926142019
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-ba8ada4-20240405150338
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.17-focal-20240909113408
	mcc-keycloak	mirantis.azurecr.io/iam/mcc-keycloak:25.0.6-20240926140203

2.28.0¶

The Mirantis Container Cloud major release 2.28.0:

Introduces support for the Cluster release 17.3.0 that is based on the Cluster release 16.3.0 and represents Mirantis OpenStack for Kubernetes (MOSK) 24.3.
Introduces support for the Cluster release 16.3.0 that is based on Mirantis Container Runtime (MCR) 23.0.14 and Mirantis Kubernetes Engine (MKE) 3.7.12 with Kubernetes 1.27.
Does not support greenfield deployments on deprecated Cluster releases of the 17.2.x and 16.2.x series. Use the latest available Cluster releases of the series instead.

Caution

Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.28.0.

Enhancements¶

This section outlines new features and enhancements introduced in the Container Cloud release 2.28.0. For the list of enhancements delivered with the Cluster releases introduced by Container Cloud 2.28.0, see 17.3.0 and 16.3.0.

General availability for Ubuntu 22.04 on MOSK clusters¶

Implemented full support for Ubuntu 22.04 LTS (Jammy Jellyfish) as the default host operating system in MOSK clusters, including greenfield deployments and update from Ubuntu 20.04 to 22.04 on existing clusters.

Ubuntu 20.04 is deprecated for greenfield deployments and supported during the MOSK 24.3 release cycle only for existing clusters.

Warning

During the course of the Container Cloud 2.28.x series, Mirantis highly recommends upgrading an operating system on any nodes of all your managed cluster machines to Ubuntu 22.04 before the next major Cluster release becomes available.

It is not mandatory to upgrade all machines at once. You can upgrade them one by one or in small batches, for example, if the maintenance window is limited in time.

Otherwise, the Cluster release update of the Ubuntu 20.04-based managed clusters will become impossible as of Container Cloud 2.29.0 with Ubuntu 22.04 as the only supported version.

Management cluster update to Container Cloud 2.29.1 will be blocked if at least one node of any related managed cluster is running Ubuntu 20.04.

Note

Since Container Cloud 2.27.0 (Cluster release 16.2.0), existing MOSK management clusters were automatically updated to Ubuntu 22.04 during cluster upgrade. Greenfield deployments of management clusters are also based on Ubuntu 22.04.

Learn more

Day-2 operations for bare metal: updating modules¶

TechPreview

Implemented the capability to update custom modules using deprecation. Once you create a new custom module, you can use it to deprecate another module by adding the deprecates field to metadata.yaml of the new module. The related HostOSConfiguration and HostOSConfigurationModules objects reflect the deprecation status of new and old modules using the corresponding fields in spec and status sections.

Also, added monitoring of deprecated modules by implementing the StackLight metrics for the Host Operating System Modules Controller along with the Day2ManagementControllerTargetDown and Day2ManagementDeprecatedConfigs alerts to notify the cloud operator about detected deprecations and issues with host-os-modules-controller.

Note

Deprecation is soft, meaning that no actual restrictions are applied to the usage of a deprecated module.

Caution

Deprecating a version automatically deprecates all lower SemVer versions of the specified module.

Learn more

Day-2 operations for bare metal: configuration enhancements for modules¶

TechPreview

Introduced the following configuration enhancements for custom modules:

Module-specific Ansible configuration
Updated the Ansible execution mechanism for running any modules. The default ansible.cfg file is now placed in /etc/ansible/mcc.cfg and used for execution of lcm-ansible and day-2 modules. However, if a module has its own ansible.cfg in the module root folder, such configuration is used for the module execution instead of the default one.
Configuration of supported operating system distribution
Added the supportedDistributions to the metadata section of a module custom resource to define the list of supported operating system distributions for the module. This field is informative and does not block the module execution on machines running non-supported distributions, but such execution will be most probably completed with an error.
Separate flag for machines requiring reboot
Introduced a separate /run/day2/reboot-required file for day-2 modules to add a notification about required reboot for a machine and a reason for reboot that appear after the module execution. The feature allows for separation of the reboot reason between LCM and day-2 operations.

Learn more

Update group for controller nodes¶

TechPreview

Implemented the update group for controller nodes using the UpdateGroup resource, which is automatically generated during initial cluster creation with the following settings:

Name: <cluster-name>-control
Index: 1
Concurrent updates: 1

This feature decouples the concurrency settings from the global cluster level and provides update flexibility.

All control plane nodes are automatically assigned to the control update group with no possibility to change it.

Note

On existing clusters created before 2.28.0 (Cluster releases 17.2.0, 16.2.0, or earlier), the control update group is created after upgrade of the Container Cloud release to 2.28.0 (Cluster release 16.3.0) on the management cluster.

Learn more

Reboot of machines using update groups¶

TechPreview

Implemented the rebootIfUpdateRequires parameter for the UpdateGroup custom resource. The parameter allows for rebooting a set of controller or worker machines added to an update group during a Cluster release update that requires a reboot, for example, when kernel version update is available in the target Cluster release. The feature reduces manual intervention and overall downtime during cluster update.

Note

By default, rebootIfUpdateRequires is set to false on managed clusters and to true on management clusters.

Learn more

Self-diagnostics for management and managed clusters¶

Implemented the Diagnostic Controller that is a tool with a set of diagnostic checks to perform self-diagnostics of any Container Cloud cluster and help the operator to easily understand, troubleshoot, and resolve potential issues against the following major subsystems: core, bare metal, Ceph, StackLight, Tungsten Fabric, and OpenStack. The Diagnostic Controller analyzes the configuration of the cluster subsystems and reports results of checks that contain useful information about cluster health.

Running self-diagnostics on both management and managed clusters is essential to ensure the overall health and optimal performance of your cluster. Mirantis recommends running self-diagnostics before cluster update, node replacement, or any other significant changes in the cluster to prevent potential issues and optimize maintenance window.

Learn more

Configuration of groups in auditd¶

TechPreview

Simplified the default auditd configuration by implementing the preset groups that you can use in presetRules instead of exact names or the virtual group all. The feature allows enabling a limited set of presets using a single keyword (group name).

Also, optimized disk usage by removing the following Docker rule that was removed from the Docker CIS Benchmark 1.3.0 due to producing excessive events:

# 1.2.4 Ensure auditing is configured for Docker files and directories - /var/lib/docker
-w /var/lib/docker -k docker

Learn more

Deployment Guide: Configure optional cluster settings - Configure Linux Audit daemon auditd

Amendments for the ClusterUpdatePlan object¶

TechPreview

Enhanced the ClusterUpdatePlan object by adding a separate update step for each UpdateGroup of worker nodes of a managed cluster. The feature allows the operator to granularly control the update process and its impact on workloads, with the option to pause the update after each step.

Also, added several StackLight alerts to notify the operator about the update progress and potential update issues.

Learn more

Refactoring of delayed auto-update of a management cluster¶

Refactored the MCCUpgrade object by implementing a new mechanism to delay Container Cloud release updates. You now have the following options for auto-update of a management cluster:

Automatically update a cluster on the publish day of a new release (by default).
Set specific days and hours for an auto-update allowing delays of up to one week. For example, if a release becomes available on Monday, you can delay it until Sunday by setting Sunday as the only permitted day for auto-updates.
Delay auto-update for minimum 20 days for each newly discovered release. The exact number of delay days is set in the release metadata and cannot be changed by the user. It depends on the specifics of each release cycle and on optional configuration of week days and hours selected for update.

You can verify the exact date of a scheduled auto-update either in the Status section of the Management Cluster Updates page in the web UI or in the status section of the MCCUpgrade object.
Combine auto-update delay with the specific days and hours setting (two previous options).

Also, optimized monitoring of auto-update by implementing several StackLight metrics for the kaas-exporter job along with the MCCUpdateBlocked and MCCUpdateScheduled alerts to notify the cloud operator about new releases as well as other important information about management cluster auto-update.

Learn more

Container Cloud web UI enhancements for the bare metal provider¶

Refactored and improved UX visibility as well as added the following functionality for the bare metal managed clusters in the Container Cloud web UI:

Reworked the Create Subnets page:
- Added the possibility to delete a subnet when it is not used by a cluster
- Changed the default value of Use whole CIDR from true to false
- Added storage subnet types: Storage access and Storage replication
Added the MetalLB Configs tab with configuration fields for MetalLB on the Networks page
Optimized the Create new machine form
Replicated the Create Credential form on the Baremetal page for easy access
Added the Labels fields to the Create L2 template and Create host profile forms as well as optimized uploading of specification data for these objects

Learn more

Documentation enhancements¶

On top of continuous improvements delivered to the existing Container Cloud guides, added documentation on how to run Ceph performance tests using Kubernetes batch or cron jobs that run fio processes according to a predefined KaaSCephOperationRequest CR.

Learn more

Operations Guide: Manage Ceph - Enable periodic Ceph performance testing

Addressed issues¶

The following issues have been addressed in the Mirantis Container Cloud release 2.28.0 along with the Cluster releases 17.3.0 and 16.3.0.

Note

This section provides descriptions of issues addressed since the last Container Cloud patch release 2.27.4.

For details on addressed issues in earlier patch releases since 2.27.0, which are also included into the major release 2.28.0, refer to 2.27.x patch releases.

[41305] [Bare metal] Fixed the issue with newly added management cluster nodes failing to undergo provisioning if the management cluster nodes were configured with a single L2 segment used for all network traffic (PXE and LCM/management networks).
[46245] [Bare metal] Fixed the issue with lack of permissions for serviceuser and users with the global-admin and operator roles to fetch HostOSConfigurationModules and HostOSConfiguration custom resources.
[43164] [StackLight] Fixed the issue with rollover policy not being added to indicies created without a policy.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.28.0 including the Cluster releases 17.3.0 and 16.3.0.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[47202] Inspection error on bare metal hosts after dnsmasq restart¶

Note

Moving forward, the workaround for this issue will be moved from Release Notes to MOSK Troubleshooting Guide: Inspection error on bare metal hosts after dnsmasq restart.

The issue can occur when:

The dnsmasq pod was moved to another node.
DHCP subnets were changed, including addition or removal. In this case, the dhcpd container of the dnsmasq pod is restarted.

Caution

If changing or adding of DHCP subnets is required to bootstrap new nodes, wait after changing or adding DHCP subnets until the dnsmasq pod becomes ready, then create BareMetalHost objects.

To verify whether the nodes are affected:

Verify whether the BareMetalHost objects contain the inspection error:

kubectl get bmh -n <managed-cluster-namespace-name>

Example of system response:

NAME            STATE         CONSUMER        ONLINE   ERROR              AGE
test-master-1   provisioned   test-master-1   true                        9d
test-master-2   provisioned   test-master-2   true                        9d
test-master-3   provisioned   test-master-3   true                        9d
test-worker-1   provisioned   test-worker-1   true                        9d
test-worker-2   provisioned   test-worker-2   true                        9d
test-worker-3   inspecting                    true     inspection error   19h

Verify whether the dnsmasq pod was in Ready state when the inspection of the affected baremetal hosts (test-worker-3 in the example above) was started:

kubectl -n kaas get pod <dnsmasq-pod-name> -oyaml

Example of system response:

...
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-10-10T15:37:34Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2024-10-11T07:38:54Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2024-10-11T07:38:54Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2024-10-10T15:37:34Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://6dbcf2fc4b36ce4c549c9191ab01f72d0236c51d42947675302675e4bfaf4cdf
    image: docker-dev-kaas-virtual.artifactory-eu.mcp.mirantis.net/bm/baremetal-dnsmasq:base-2-28-alpine-20240812132650
    imageID: docker-dev-kaas-virtual.artifactory-eu.mcp.mirantis.net/bm/baremetal-dnsmasq@sha256:3dad3e278add18e69b2608e462691c4823942641a0f0e25e6811e703e3c23b3b
    lastState:
      terminated:
        containerID: containerd://816fcf079cd544acd74e312065de5b5ed4dbf1dc6159fefffff4f644b5e45987
        exitCode: 0
        finishedAt: "2024-10-11T07:38:35Z"
        reason: Completed
        startedAt: "2024-10-10T15:37:45Z"
    name: dhcpd
    ready: true
    restartCount: 2
    started: true
    state:
      running:
        startedAt: "2024-10-11T07:38:37Z"
  ...

In the system response above, the dhcpd container was not ready between "2024-10-11T07:38:35Z" and "2024-10-11T07:38:54Z".

Verify the affected baremetal host. For example:

kubectl get bmh -n managed-ns test-worker-3 -oyaml

Example of system response:

...
status:
  errorCount: 15
  errorMessage: Introspection timeout
  errorType: inspection error
  ...
  operationHistory:
    deprovision:
      end: null
      start: null
    inspect:
      end: null
      start: "2024-10-11T07:38:19Z"
    provision:
      end: null
      start: null
    register:
      end: "2024-10-11T07:38:19Z"
      start: "2024-10-11T07:37:25Z"

Workaround

Reboot the node using the IPMI reset or cycle command.
If the node fails to boot, remove the failed BareMetalHost object and create it again:
1. Remove BareMetalHost object. For example:
```
kubectl delete bmh -n managed-ns test-worker-3
```
2. Verify that the BareMetalHost object is removed:
```
kubectl get bmh -n managed-ns test-worker-3
```
3. Create a BareMetalHost object from the template. For example:
```
kubectl create -f bmhc-test-worker-3.yaml
kubectl create -f bmh-test-worker-3.yaml
```

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

Ceph¶

[50566] Ceph upgrade is very slow during patch or major cluster update¶

Fixed in 2.29.3 (17.3.8, 16.3.8, and 16.4.3)

 Warning  Unhealthy  57s (x16 over 3m27s)  kubelet  Startup probe failed:
 ceph daemon health check failed with the following output:
> no valid command found; 10 closest matches:
> 0
> 1
> 2
> abort
> assert
> bluefs debug_inject_read_zeros
> bluefs files list
> bluefs stats
> bluestore bluefs device info [<alloc_size:int>]
> config diff
> admin_socket: invalid command

Workaround:

Complete the following steps during every patch or major cluster update of the Cluster releases 17.2.x, 17.3.x, and 17.4.x (until Ceph 18.2.5 becomes supported):

Plan extra time in the maintenance window for the patch cluster update.

Slow starts will still impact the update procedure, but after completing the following step, the recovery process noticeably shortens without affecting the overall cluster state and data responsiveness.
Select one of the following options:
- Before the cluster update, set the noout flag:
```
ceph osd set noout
```
  Once the Ceph OSDs image upgrade is done, unset the flag:
```
ceph osd unset noout
```
- Monitor the Ceph OSDs image upgrade. If the symptoms of slow start appear, set the noout flag as soon as possible. Once the Ceph OSDs image upgrade is done, unset the flag.

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

LCM¶

[46808] Old kernel metapackages are not removed during kernel upgrade¶

Fixed in 2.28.1 (17.2.5, 16.2.5, and 16.3.1)

After upgrade of kernel to the latest supported version, old kernel metapackages may remain on the cluster. The issue occurs if the system kernel line is changed from LTS to HWE. This setting is controlled by the upgrade_kernel_version parameter located in the ClusterRelease object under the deploy StateItem. As a result, the operating system has both LTS and HWE kernel packages installed and regularly updated, but only one kernel image is used (loaded into memory). The unused kernel images consume minimal amount of disk space.

Therefore, you can safely disregard the issue because it does not affect cluster operability. If you still require removing unused kernel metapackages, contact Mirantis support for detailed instructions.

[39437] Failure to replace a master node on a Container Cloud cluster¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

During the replacement of a master node on a cluster of any type, the process may get stuck with Kubelet's NodeReady condition is Unknown in the machine status on the remaining master nodes.

As a workaround, log in on the affected node and run the following command:

docker restart ucp-kubelet

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

[30294] Replacement of a master node is stuck on the calico-node Pod start¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a master node on a cluster of any type, the calico-node Pod fails to start on a new node that has the same IP address as the node being replaced.

Workaround:

From a CLI with an MKE client bundle, create a shell alias to start calicoctl using the mirantis/ucp-dsinfo image:

Since MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/key.pem \
-e ETCD_CA_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/ca.pem \
-e ETCD_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/cert.pem \
-v /var/run/calico:/var/run/calico \
-v /var/lib/docker/volumes/ucp-kv-certs/_data:/var/lib/docker/volumes/ucp-kv-certs/_data:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl \
"

Before MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
-e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
-e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
-v /var/run/calico:/var/run/calico \
-v ucp-node-certs:/ucp-node-certs:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl --allow-version-mismatch \
"

In the above command, replace the following values with the corresponding settings of the affected cluster:

<etcdEndpoint> is the etcd endpoint defined in the Calico configuration file. For example, ETCD_ENDPOINTS=127.0.0.1:12378
<mkeVersion> is the MKE version installed on your cluster. For example, mirantis/ucp-dsinfo:3.5.7.

Verify the node list on the cluster:
```
kubectl get node
```
Compare this list with the node list in Calico to identify the old node:
```
calicoctl get node -o wide
```

Remove the old node from Calico:

calicoctl delete node kaas-node-<nodeID>

[5782] Manager machine fails to be deployed during node replacement¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a manager machine, the following problems may occur:

The system adds the node to Docker swarm but not to Kubernetes
The node Deployment gets stuck with failed RethinkDB health checks

Workaround:

Delete the failed node.
Wait for the MKE cluster to become healthy. To monitor the cluster status:
1. Log in to the MKE web UI as described in Connect to the Mirantis Kubernetes Engine web UI.
2. Monitor the cluster status as described in MKE Operations Guide: Monitor an MKE cluster with the MKE web UI.
Deploy a new node.

[5568] The calico-kube-controllers Pod fails to clean up resources¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During the unsafe or forced deletion of a manager machine running the calico-kube-controllers Pod in the kube-system namespace, the following issues occur:

The calico-kube-controllers Pod fails to clean up resources associated with the deleted node
The calico-node Pod may fail to start up on a newly created node if the machine is provisioned with the same IP address as the deleted machine had

As a workaround, before deletion of the node running the calico-kube-controllers Pod, cordon and drain the node:

kubectl cordon <nodeName>
kubectl drain <nodeName>

StackLight¶

[47594] Patroni pods may get stuck in the CrashLoopBackOff state¶

Fixed in 2.28.3 (17.2.7, 16.2.7, and 16.3.3)

The Patroni pods may get stuck in the CrashLoopBackOff state due to the patroni container being terminated with reason: OOMKilled that you can see in the pod status. For example:

kubectl get pod/patroni-13-0 -n stacklight -o yaml

...
  - containerID: docker://<ID>`
    image: mirantis.azurecr.io/stacklight/spilo:13-2.1p9-20240828023010
    imageID: docker-pullable://mirantis.azurecr.io/stacklight/spilo@sha256:<ID>
    lastState:
      terminated:
        containerID: docker://<ID>
        exitCode: 137
        finishedAt: "2024-10-17T14:26:25Z"
        reason: OOMKilled
        startedAt: "2024-10-17T14:23:25Z"
    name: patroni
...

As a workaround, increase the memory limit for PostgreSQL to 20Gi in the Cluster object:

spec:
  providerSpec:
    value:
      helmReleases:
      - name: stacklight
        values:
          resources:
            postgresql:
              limits:
                memory: "20Gi"

[47304] OpenSearch does not store kubelet logs¶

Fixed in 2.28.2 (17.2.6, 16.2.6, and 16.3.2)

Due to the JSON-based format of ucp-kubelet logs, OpenSearch does not store kubelet logs. Mirantis is working on the issue and will deliver the resolution in one of the nearest patch releases.

[44193] OpenSearch reaches 85% disk usage watermark affecting the cluster state¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

Caution

The issue and the provided workaround apply only for clusters on which OpenSearch and Prometheus utilize the same storage pool.

To verify that the cluster is affected:

Verify the result of the following formula:
```
0.8 × OpenSearch_PVC_Size_GB + Prometheus_PVC_Size_GB > 0.85 × Total_Storage_Capacity_GB
```
In the formula, define the following values:
OpenSearch_PVC_Size_GB
Derived from .values.elasticsearch.persistentVolumeUsableStorageSizeGB, defaulting to .values.elasticsearch.persistentVolumeClaimSize if unspecified. To obtain the OpenSearch PVC size:
kubectl -n <namespaceName> get cluster <clusterName> -o yaml |\ yq '.spec.providerSpec.value.helmReleases[] | select(.name == "stacklight") | .values.elasticsearch.persistentVolumeClaimSize '
Example of system response:
10000Gi
Prometheus_PVC_Size_GB
Sourced from .values.prometheusServer.persistentVolumeClaimSize. To obtain the Prometheus PVC size:
kubectl -n <namespaceName> get cluster <clusterName> -o yaml |\ yq '.spec.providerSpec.value.helmReleases[] | select(.name == "stacklight") | .values.prometheusServer.persistentVolumeClaimSize '
Example of system response:
4000Gi
Total_Storage_Capacity_GB
Total capacity of the OpenSearch PVCs. For LVP, the capacity of the storage pool. To obtain the total capacity:
kubectl get pvc -n stacklight -l app=opensearch-master \ -o custom-columns=NAME:.metadata.name,CAPACITY:.status.capacity.storage
The system response contains multiple outputs, one per opensearch-master node. Select the capacity for the affected node.
Note

Convert the values to GB if they are set in different units.

If the formula result is positive, it is an early indication that the cluster is affected.
Verify whether the OpenSearchClusterStatusWarning or OpenSearchClusterStatusCritical alert is firing. And if so, verify the following:
1. Log in to the OpenSearch web UI.
2. In Management -> Dev Tools, run the following command:
```
GET _cluster/allocation/explain
```
  The following system response indicates that the corresponding node is affected:
```
"explanation": "the node is above the low watermark cluster setting \
[cluster.routing.allocation.disk.watermark.low=85%], using more disk space \
than the maximum allowed [85.0%], actual free: [xx.xxx%]"
```
  Note
  
  The system response may contain even higher watermark percent than 85.0%, depending on the case.

Workaround:

Warning

The workaround implies adjustement of the retention threshold for OpenSearch. And depending on the new threshold, some old logs will be deleted.

Adjust or set .values.elasticsearch.persistentVolumeUsableStorageSizeGB to a lower value for the affection check formula to be non-positive. For configuration details, see MOSK Operations Guide: StackLight configuration parameters - OpenSearch.

Mirantis also recommends reserving some space for other PVCs using storage from the pool. Use the following formula to calculate the required space:
```
persistentVolumeUsableStorageSizeGB =
0.84 × ((1 - Reserved_Percentage - Filesystem_Reserve) ×
Total_Storage_Capacity_GB - Prometheus_PVC_Size_GB) /
0.8
```
In the formula, define the following values:
Reserved_Percentage
A user-defined variable that specifies what percentage of the total storage capacity should not be used by OpenSearch or Prometheus. This is used to reserve space for other components. It should be expressed as a decimal. For example, for 5% of reservation, Reserved_Percentage is 0.05. Mirantis recommends using 0.05 as a starting point.

Filesystem_Reserve
Percentage to deduct for filesystems that may reserve some portion of the available storage, which is marked as occupied. For example, for EXT4, it is 5% by default, so the value must be 0.05.

Prometheus_PVC_Size_GB
Sourced from .values.prometheusServer.persistentVolumeClaimSize.

Total_Storage_Capacity_GB
Total capacity of the OpenSearch PVCs. For LVP, the capacity of the storage pool. To obtain the total capacity:
kubectl get pvc -n stacklight -l app=opensearch-master \ -o custom-columns=NAME:.metadata.name,CAPACITY:.status.capacity.storage
The system response contains multiple outputs, one per opensearch-master node. Select the capacity for the affected node.
Note

Convert the values to GB if they are set in different units.

Calculation of above formula provides a maximum safe storage to allocate for .values.elasticsearch.persistentVolumeUsableStorageSizeGB. Use this formula as a reference for setting .values.elasticsearch.persistentVolumeUsableStorageSizeGB on a cluster.
Wait up to 15-20 mins for OpenSearch to perform the cleaning.
Verify that the cluster is not affected anymore using the procedure above.

Container Cloud web UI¶

[50181] Failure to deploy a compact cluster¶

To work around the issue, manually add the required labels using CLI. Once done, the cluster deployment resumes.

[50168] Inability to use a new project right after creation¶

A newly created project does not display all available tabs in the Container Cloud web UI and contains different access denied errors during first five minutes after creation.

To work around the issue, refresh the browser in five minutes after the project creation.

Components versions¶

The following table lists the major components and their versions delivered in Container Cloud 2.28.0. The components that are newly added, updated, deprecated, or removed as compared to 2.27.0, are marked with a corresponding superscript, for example, admission-controller ^Updated.

Component	Application/Service	Version
Bare metal	baremetal-dnsmasq ^Updated	base-2-28-alpine-20240906160120
	baremetal-operator ^Updated	base-2-28-alpine-20240910093836
	baremetal-provider ^Updated	1.41.14
	bm-collective ^Updated	base-2-28-alpine-20240910093747
	cluster-api-provider-baremetal ^Updated	1.41.14
	ironic ^Updated	antelope-jammy-20240716113922
	ironic-inspector ^Updated	antelope-jammy-20240716113922
	ironic-prometheus-exporter	0.1-20240819102310
	kaas-ipam ^Updated	base-2-28-alpine-20240910095249
	kubernetes-entrypoint	v1.0.1-4e381cb-20240813170642
	mariadb	10.6.17-focal-20240523075821
	metallb-controller ^Updated	v0.14.5-ed177720-amd64
	metallb-speaker ^Updated	v0.14.5-ed177720-amd64
	syslog-ng ^Updated	base-alpine-20240906155734
Container Cloud	admission-controller ^Updated	1.41.14
	agent-controller ^Updated	1.41.14
	byo-cluster-api-controller ^Updated	1.41.14
	byo-credentials-controller ^Removed	n/a
	ceph-kcc-controller ^Updated	1.41.14
	cert-manager-controller ^Updated	1.11.0-8
	configuration-collector ^Updated	1.41.14
	event-controller ^Updated	1.41.14
	frontend ^Updated	1.41.14
	golang	1.22.7
	iam-controller ^Updated	1.41.14
	kaas-exporter ^Updated	1.41.14
	kproxy ^Updated	1.41.14
	lcm-controller ^Updated	1.41.14
	license-controller ^Updated	1.41.14
	machinepool-controller ^Updated	1.41.14
	mcc-haproxy ^Updated	0.26.0-95-g95f0130
	nginx ^Updated	1.41.14
	portforward-controller ^Updated	1.41.14
	proxy-controller ^Updated	1.41.14
	rbac-controller ^Updated	1.41.14
	registry ^Updated	2.8.1-13
	release-controller ^Updated	1.41.14
	rhellicense-controller ^Removed	n/a
	scope-controller ^Updated	1.41.14
	secret-controller ^Updated	1.41.14
	storage-discovery ^Updated	1.41.14
	user-controller ^Updated	1.41.14
IAM ^Updated	iam	1.41.14
	mariadb	10.6.17-focal-20240909113408
	mcc-keycloak ^Updated	25.0.6-20240926140203
OpenStack ^Updated	host-os-modules-controller	1.41.14
	openstack-cluster-api-controller	1.41.14
	openstack-provider	1.41.14
	os-credentials-controller ^Removed	n/a

Artifacts¶

This section lists the artifacts of components included in the Container Cloud release 2.27.0. The components that are newly added, updated, deprecated, or removed as compared to 2.27.0, are marked with a corresponding superscript, for example, admission-controller ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries ^Updated	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20240911112529
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20240911112529
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.41.14.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.41.14.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.41.14.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.41.14.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.41.14.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.41.14.tgz
Docker images	ambasador ^Updated	mirantis.azurecr.io/core/external/nginx:1.41.14
	baremetal-dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-28-alpine-20240906160120
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-2-28-alpine-20240910093836
	bm-collective ^Updated	mirantis.azurecr.io/bm/bm-collective:base-2-28-alpine-20240910093747
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.41.14
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:antelope-jammy-20240716113922
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:antelope-jammy-20240716113922
	ironic-prometheus-exporter ^Updated	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20240819102310
	kaas-ipam ^Updated	mirantis.azurecr.io/bm/kaas-ipam:base-2-28-alpine-20240910095249
	kubernetes-entrypoint ^Updated	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-4e381cb-20240813170642
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.17-focal-20240523075821
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.26.0-95-g95f0130
	metallb-controller ^Updated	mirantis.azurecr.io/bm/metallb/controller:v0.14.5-ed177720-amd64
	metallb-speaker ^Updated	mirantis.azurecr.io/bm/metallb/speaker:v0.14.5-ed177720-amd64
	syslog-ng ^Updated	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20240906155734

Core artifacts¶

Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.41.14.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.41.14.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.41.14.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.41.14.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.41.14.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.41.14.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.41.14.tgz
	credentials-controller ^New	https://binary.mirantis.com/core/helm/credentials-controller-1.41.14.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.41.14.tgz
	host-os-modules-controller	https://binary.mirantis.com/core/helm/host-os-modules-controller-1.41.14.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.41.14.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.41.14.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.41.14.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.41.14.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.41.14.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.41.14.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.41.14.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.41.14.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.41.14.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.41.14.tgz
	os-credentials-controller ^Removed	n/a
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.41.14.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.41.14.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.41.14.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.41.14.tgz
	secret-controller	https://binary.mirantis.com/core/helm/secret-controller-1.41.14.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.41.14.tgz
Docker images ^Updated	admission-controller	mirantis.azurecr.io/core/admission-controller:1.41.14
	agent-controller	mirantis.azurecr.io/core/agent-controller:1.41.14
	ceph-kcc-controller	mirantis.azurecr.io/core/ceph-kcc-controller:1.41.14
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-8
	configuration-collector	mirantis.azurecr.io/core/configuration-collector:1.41.14
	credentials-controller ^New	mirantis.azurecr.io/core/credentials-controller:1.41.14
	event-controller	mirantis.azurecr.io/core/event-controller:1.41.14
	frontend	mirantis.azurecr.io/core/frontend:1.41.14
	host-os-modules-controller	mirantis.azurecr.io/core/host-os-modules-controller:1.41.14
	iam-controller	mirantis.azurecr.io/core/iam-controller:1.41.14
	kaas-exporter	mirantis.azurecr.io/core/kaas-exporter:1.41.14
	kproxy	mirantis.azurecr.io/core/kproxy:1.41.14
	lcm-controller	mirantis.azurecr.io/core/lcm-controller:1.41.14
	license-controller	mirantis.azurecr.io/core/license-controller:1.41.14
	machinepool-controller	mirantis.azurecr.io/core/machinepool-controller:1.41.14
	mcc-cache-warmup	mirantis.azurecr.io/core/mcc-cache-warmup:1.41.14
	mcc-haproxy	mirantis.azurecr.io/lcm/mcc-haproxy:v0.26.0-95-g95f0130
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.26.0-95-g95f0130
	nginx	mirantis.azurecr.io/core/external/nginx:1.41.14
	openstack-cluster-api-controller	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.41.14
	os-credentials-controller ^Removed	n/a
	portforward-controller	mirantis.azurecr.io/core/portforward-controller:1.41.14
	rbac-controller	mirantis.azurecr.io/core/rbac-controller:1.41.14
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-13
	release-controller	mirantis.azurecr.io/core/release-controller:1.41.14
	scope-controller	mirantis.azurecr.io/core/scope-controller:1.41.14
	secret-controller	mirantis.azurecr.io/core/secret-controller:1.41.14
	user-controller	mirantis.azurecr.io/core/user-controller:1.41.14

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts ^Updated	iam	https://binary.mirantis.com/core/helm/iam-1.41.14.tgz
Docker images	kubectl ^Updated	mirantis.azurecr.io/general/kubectl:20240926142019
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-ba8ada4-20240405150338
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.17-focal-20240909113408
	mcc-keycloak ^Updated	mirantis.azurecr.io/iam/mcc-keycloak:25.0.6-20240926140203

Security notes¶

In total, since Container Cloud 2.27.0, in 2.28.0, 2614 Common Vulnerabilities and Exposures (CVE) have been fixed: 299 of critical and 2315 of high severity.

The table below includes the total numbers of addressed unique and common vulnerabilities and exposures (CVE) by product component since the 2.27.4 patch release. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	Critical	High	Total
Ceph	Unique	0	5	5
Ceph	Common	0	211	211
KaaS core	Unique	4	11	15
KaaS core	Common	10	315	325
StackLight	Unique	1	7	8
StackLight	Common	1	25	26

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 24.3: Security notes.

Update notes¶

This section describes the specific actions you as a cloud operator need to complete before or after your Container Cloud cluster update to the Cluster releases 17.3.0 or 16.3.0.

Consider this information as a supplement to the generic update procedures published in Operations Guide: Automatic upgrade of a management cluster and Update a managed cluster.

Pre-update actions¶

Change label values in Ceph metrics used in customizations¶

Note

If you do not use Ceph metrics in any customizations, for example, custom alerts, Grafana dashboards, or queries in custom workloads, skip this section.

After deprecating the performance metric exporter that is integrated into the Ceph Manager daemon for the sake of the dedicated Ceph Exporter daemon in Container Cloud 2.27.0, you may need to update values of several labels in Ceph metrics if you use them in any customizations such as custom alerts, Grafana dashboards, or queries in custom tools. These labels are changed in Container Cloud 2.28.0 (Cluster releases 16.3.0 and 17.3.0).

Note

Names of metrics are changed, no metrics are removed.

All Ceph metrics to be collected by the Ceph Exporter daemon changed their labels job and instance due to scraping metrics from new Ceph Exporter daemon instead of the performance metric exporter of Ceph Manager:

Values of the job labels are changed from rook-ceph-mgr to prometheus-rook-exporter for all Ceph metrics moved to Ceph Exporter. The full list of moved metrics is presented below.
Values of the instance labels are changed from the metric endpoint of Ceph Manager with port 9283 to the metric endpoint of Ceph Exporter with port 9926 for all Ceph metrics moved to Ceph Exporter. The full list of moved metrics is presented below.
Values of the instance_id labels of Ceph metrics from the RADOS Gateway (RGW) daemons are changed from the daemon GID to the daemon subname. For example, instead of instance_id="<RGW_PROCESS_GID>", the instance_id="a" (ceph_rgw_qlen{instance_id="a"}) is now used. The list of moved Ceph RGW metrics is presented below.

Post-update actions¶

Manually disable collection of performance metrics by Ceph Manager (optional)¶

Since Container Cloud 2.28.0 (Cluster releases 17.3.0 and 16.3.0), Ceph cluster metrics are collected by the dedicated Ceph Exporter daemon. At the same time, same metrics are still available to be collected by the Ceph Manager daemon. To improve performance of the Ceph Manager daemon, you can manually disable collection of performance metrics by Ceph Manager, which are collected by the Ceph Exporter daemon.

To disable performance metrics for the Ceph Manager daemon, add the following parameter to the KaaSCephCluster spec in the rookConfig section:

spec:
  cephClusterSpec:
    rookConfig:
      "mgr|mgr/prometheus/exclude_perf_counters": "true"

Once you add this option, Ceph performance metrics are collected by the Ceph Exporter daemon only. For more details, see Official Ceph documentation.

Upgrade to Ubuntu 22.04 on baremetal-based clusters¶

In Container Cloud 2.29.0, the Cluster release update of the Ubuntu 20.04-based managed clusters will become impossible, and Ubuntu 22.04 will become the only supported version of the operating system. Therefore, ensure that every node of your managed clusters are running Ubuntu 22.04 to unblock managed cluster update in Container Cloud 2.29.0.

Warning

Management cluster update to Container Cloud 2.29.1 will be blocked if at least one node of any related managed cluster is running Ubuntu 20.04.

Therefore, if your existing cluster runs nodes on Ubuntu 20.04, prevent blocking of your cluster update by upgrading all cluster nodes to Ubuntu 22.04 during the course of the Container Cloud 2.28.x series. For the update procedure, refer to Mirantis OpenStack for Kubernetes documentation: Bare metal operations - Upgrade an operating system distribution.

It is not mandatory to upgrade all machines at once. You can upgrade them one by one or in small batches, for example, if the maintenance window is limited in time.

Note

Warning

Usage of third-party software, which is not part of Mirantis-supported configurations, for example, the use of custom DPDK modules, may block upgrade of an operating system distribution. Users are fully responsible for ensuring the compatibility of such custom components with the latest supported Ubuntu version.

See also

2.27.4¶

Note

For MOSK clusters, Container Cloud 2.27.4 is the second patch release of MOSK 24.2.x series using the patch Cluster release 17.2.4. For the update path of 24.1 and 24.2 series, see MOSK documentation: Cluster update scheme.

The Container Cloud patch release 2.27.4, which is based on the 2.27.0 major release, provides the following updates:

Support for the patch Cluster releases 16.2.4 and 17.2.4 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 24.2.2.
Bare metal: update of Ubuntu mirror from ubuntu-2024-08-06-014502 to ubuntu-2024-08-21-014714 along with update of the minor kernel version from 5.15.0-117-generic to 5.15.0-119-generic for Jammy and to 5.15.0-118-generic for Focal.
Security fixes for CVEs in images.

This patch release also supports the latest major Cluster releases 17.2.0 and 16.2.0. And it does not support greenfield deployments based on deprecated Cluster releases. Use the latest available Cluster release instead.

For main deliverables of the parent Container Cloud release of 2.27.4, refer to 2.27.0.

Security notes¶

In total, since Container Cloud 2.27.3, 131 Common Vulnerabilities and Exposures (CVE) have been fixed in 2.27.4: 15 of critical and 116 of high severity.

The table below includes the total numbers of addressed unique and common CVEs in images by product component since Container Cloud 2.27.3. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	Critical	High	Total
Ceph	Unique	0	1	1
Ceph	Common	0	3	3
Kaas core	Unique	3	19	22
Kaas core	Common	14	105	119
StackLight	Unique	1	8	9
StackLight	Common	1	8	9

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 24.2.2: Security notes.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.27.4 including the Cluster releases 16.2.4 and 17.2.4.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[47202] Inspection error on bare metal hosts after dnsmasq restart¶

Note

Moving forward, the workaround for this issue will be moved from Release Notes to MOSK Troubleshooting Guide: Inspection error on bare metal hosts after dnsmasq restart.

The issue can occur when:

The dnsmasq pod was moved to another node.
DHCP subnets were changed, including addition or removal. In this case, the dhcpd container of the dnsmasq pod is restarted.

Caution

If changing or adding of DHCP subnets is required to bootstrap new nodes, wait after changing or adding DHCP subnets until the dnsmasq pod becomes ready, then create BareMetalHost objects.

To verify whether the nodes are affected:

Verify whether the BareMetalHost objects contain the inspection error:

kubectl get bmh -n <managed-cluster-namespace-name>

Example of system response:

NAME            STATE         CONSUMER        ONLINE   ERROR              AGE
test-master-1   provisioned   test-master-1   true                        9d
test-master-2   provisioned   test-master-2   true                        9d
test-master-3   provisioned   test-master-3   true                        9d
test-worker-1   provisioned   test-worker-1   true                        9d
test-worker-2   provisioned   test-worker-2   true                        9d
test-worker-3   inspecting                    true     inspection error   19h

Verify whether the dnsmasq pod was in Ready state when the inspection of the affected baremetal hosts (test-worker-3 in the example above) was started:

kubectl -n kaas get pod <dnsmasq-pod-name> -oyaml

Example of system response:

...
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-10-10T15:37:34Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2024-10-11T07:38:54Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2024-10-11T07:38:54Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2024-10-10T15:37:34Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://6dbcf2fc4b36ce4c549c9191ab01f72d0236c51d42947675302675e4bfaf4cdf
    image: docker-dev-kaas-virtual.artifactory-eu.mcp.mirantis.net/bm/baremetal-dnsmasq:base-2-28-alpine-20240812132650
    imageID: docker-dev-kaas-virtual.artifactory-eu.mcp.mirantis.net/bm/baremetal-dnsmasq@sha256:3dad3e278add18e69b2608e462691c4823942641a0f0e25e6811e703e3c23b3b
    lastState:
      terminated:
        containerID: containerd://816fcf079cd544acd74e312065de5b5ed4dbf1dc6159fefffff4f644b5e45987
        exitCode: 0
        finishedAt: "2024-10-11T07:38:35Z"
        reason: Completed
        startedAt: "2024-10-10T15:37:45Z"
    name: dhcpd
    ready: true
    restartCount: 2
    started: true
    state:
      running:
        startedAt: "2024-10-11T07:38:37Z"
  ...

In the system response above, the dhcpd container was not ready between "2024-10-11T07:38:35Z" and "2024-10-11T07:38:54Z".

Verify the affected baremetal host. For example:

kubectl get bmh -n managed-ns test-worker-3 -oyaml

Example of system response:

...
status:
  errorCount: 15
  errorMessage: Introspection timeout
  errorType: inspection error
  ...
  operationHistory:
    deprovision:
      end: null
      start: null
    inspect:
      end: null
      start: "2024-10-11T07:38:19Z"
    provision:
      end: null
      start: null
    register:
      end: "2024-10-11T07:38:19Z"
      start: "2024-10-11T07:37:25Z"

Workaround

Reboot the node using the IPMI reset or cycle command.
If the node fails to boot, remove the failed BareMetalHost object and create it again:
1. Remove BareMetalHost object. For example:
```
kubectl delete bmh -n managed-ns test-worker-3
```
2. Verify that the BareMetalHost object is removed:
```
kubectl get bmh -n managed-ns test-worker-3
```
3. Create a BareMetalHost object from the template. For example:
```
kubectl create -f bmhc-test-worker-3.yaml
kubectl create -f bmh-test-worker-3.yaml
```

[46245] Lack of access permissions for HOC and HOCM objects¶

Fixed in 2.28.0 (17.3.0 and 16.3.0)

When trying to list the HostOSConfigurationModules and HostOSConfiguration custom resources, serviceuser or a user with the global-admin or operator role obtains the access denied error. For example:

kubectl --kubeconfig ~/.kube/mgmt-config get hocm

Error from server (Forbidden): hostosconfigurationmodules.kaas.mirantis.com is forbidden:
User "2d74348b-5669-4c65-af31-6c05dbedac5f" cannot list resource "hostosconfigurationmodules"
in API group "kaas.mirantis.com" at the cluster scope: access denied

Workaround:

Modify the global-admin role by adding a new entry with the following contents to the rules list:

kubectl edit clusterroles kaas-global-admin

- apiGroups: [kaas.mirantis.com]
  resources: [hostosconfigurationmodules]
  verbs: ['*']

For each Container Cloud project, modify the kaas-operator role by adding a new entry with the following contents to the rules list:

kubectl -n <projectName> edit roles kaas-operator

- apiGroups: [kaas.mirantis.com]
  resources: [hostosconfigurations]
  verbs: ['*']

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[41305] DHCP responses are lost between dnsmasq and dhcp-relay pods¶

Fixed in 2.28.0 (17.3.0 and 16.3.0)

After node maintenance of a management cluster, the newly added nodes may fail to undergo provisioning successfully. The issue relates to new nodes that are in the same L2 domain as the management cluster.

The issue was observed on environments having management cluster nodes configured with a single L2 segment used for all network traffic (PXE and LCM/management networks).

To verify whether the cluster is affected:

Verify whether the dnsmasq and dhcp-relay pods run on the same node in the management cluster:

kubectl -n kaas get pods -o wide| grep -e "dhcp\|dnsmasq"

Example of system response:

dhcp-relay-7d85f75f76-5vdw2   2/2   Running   2 (36h ago)   36h   10.10.0.122     kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>
dnsmasq-8f4b484b4-slhbd       5/5   Running   1 (36h ago)   36h   10.233.123.75   kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>

If this is the case, proceed to the workaround below.

Workaround:

Make sure that at least two management cluster nodes are schedulable:

kubectl get node

Example of a positive system response:

NAME                                             STATUS   ROLES    AGE   VERSION
kaas-node-bcedb87b-b3ce-46a4-a4ca-ea3068689e40   Ready    master   37h   v1.27.10-mirantis-1
kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   Ready    master   37h   v1.27.10-mirantis-1
kaas-node-ad5a6f51-b98f-43c3-91d5-55fed3d0ff21   Ready    master   37h   v1.27.10-mirantis-1

Delete the dhcp-relay pod:

kubectl -n kaas delete pod <dhcp-relay-xxxxx>

Verify that the dnsmasq and dhcp-relay pods are scheduled into different nodes:

kubectl -n kaas get pods -o wide| grep -e "dhcp\|dnsmasq"

Example of a positive system response:

dhcp-relay-7d85f75f76-rkv03   2/2   Running   0             49s   10.10.0.121     kaas-node-bcedb87b-b3ce-46a4-a4ca-ea3068689e40   <none>   <none>
dnsmasq-8f4b484b4-slhbd       5/5   Running   1 (37h ago)   37h   10.233.123.75   kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

LCM¶

[39437] Failure to replace a master node on a Container Cloud cluster¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

During the replacement of a master node on a cluster of any type, the process may get stuck with Kubelet's NodeReady condition is Unknown in the machine status on the remaining master nodes.

As a workaround, log in on the affected node and run the following command:

docker restart ucp-kubelet

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

[30294] Replacement of a master node is stuck on the calico-node Pod start¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a master node on a cluster of any type, the calico-node Pod fails to start on a new node that has the same IP address as the node being replaced.

Workaround:

From a CLI with an MKE client bundle, create a shell alias to start calicoctl using the mirantis/ucp-dsinfo image:

Since MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/key.pem \
-e ETCD_CA_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/ca.pem \
-e ETCD_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/cert.pem \
-v /var/run/calico:/var/run/calico \
-v /var/lib/docker/volumes/ucp-kv-certs/_data:/var/lib/docker/volumes/ucp-kv-certs/_data:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl \
"

Before MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
-e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
-e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
-v /var/run/calico:/var/run/calico \
-v ucp-node-certs:/ucp-node-certs:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl --allow-version-mismatch \
"

In the above command, replace the following values with the corresponding settings of the affected cluster:

<etcdEndpoint> is the etcd endpoint defined in the Calico configuration file. For example, ETCD_ENDPOINTS=127.0.0.1:12378
<mkeVersion> is the MKE version installed on your cluster. For example, mirantis/ucp-dsinfo:3.5.7.

Verify the node list on the cluster:
```
kubectl get node
```
Compare this list with the node list in Calico to identify the old node:
```
calicoctl get node -o wide
```

Remove the old node from Calico:

calicoctl delete node kaas-node-<nodeID>

[5782] Manager machine fails to be deployed during node replacement¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a manager machine, the following problems may occur:

The system adds the node to Docker swarm but not to Kubernetes
The node Deployment gets stuck with failed RethinkDB health checks

Workaround:

Delete the failed node.
Wait for the MKE cluster to become healthy. To monitor the cluster status:
1. Log in to the MKE web UI as described in Connect to the Mirantis Kubernetes Engine web UI.
2. Monitor the cluster status as described in MKE Operations Guide: Monitor an MKE cluster with the MKE web UI.
Deploy a new node.

[5568] The calico-kube-controllers Pod fails to clean up resources¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During the unsafe or forced deletion of a manager machine running the calico-kube-controllers Pod in the kube-system namespace, the following issues occur:

The calico-kube-controllers Pod fails to clean up resources associated with the deleted node
The calico-node Pod may fail to start up on a newly created node if the machine is provisioned with the same IP address as the deleted machine had

As a workaround, before deletion of the node running the calico-kube-controllers Pod, cordon and drain the node:

kubectl cordon <nodeName>
kubectl drain <nodeName>

Ceph¶

[50566] Ceph upgrade is very slow during patch or major cluster update¶

Fixed in 2.29.3 (17.3.8, 16.3.8, and 16.4.3)

 Warning  Unhealthy  57s (x16 over 3m27s)  kubelet  Startup probe failed:
 ceph daemon health check failed with the following output:
> no valid command found; 10 closest matches:
> 0
> 1
> 2
> abort
> assert
> bluefs debug_inject_read_zeros
> bluefs files list
> bluefs stats
> bluestore bluefs device info [<alloc_size:int>]
> config diff
> admin_socket: invalid command

Workaround:

Complete the following steps during every patch or major cluster update of the Cluster releases 17.2.x, 17.3.x, and 17.4.x (until Ceph 18.2.5 becomes supported):

Plan extra time in the maintenance window for the patch cluster update.

Slow starts will still impact the update procedure, but after completing the following step, the recovery process noticeably shortens without affecting the overall cluster state and data responsiveness.
Select one of the following options:
- Before the cluster update, set the noout flag:
```
ceph osd set noout
```
  Once the Ceph OSDs image upgrade is done, unset the flag:
```
ceph osd unset noout
```
- Monitor the Ceph OSDs image upgrade. If the symptoms of slow start appear, set the noout flag as soon as possible. Once the Ceph OSDs image upgrade is done, unset the flag.

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

Container Cloud web UI¶

[50181] Failure to deploy a compact cluster¶

To work around the issue, manually add the required labels using CLI. Once done, the cluster deployment resumes.

[50168] Inability to use a new project right after creation¶

A newly created project does not display all available tabs in the Container Cloud web UI and contains different access denied errors during first five minutes after creation.

To work around the issue, refresh the browser in five minutes after the project creation.

Patch cluster update¶

[49713] Patch update is stuck with some nodes in Prepare state¶

Patch update from 2.27.3 to 2.27.4 may get stuck with one or more management cluster nodes remaining in the Prepare state and with the following error in the lcm-controller logs on the management cluster:

failed to create cluster updater for cluster default/kaas-mgmt:
machine update group not found for machine default/master-0

To work around the issue, in the LCMMachine objects of the management cluster, set the following annotation:

lcm.mirantis.com/update-group: <mgmt cluster name>-controlplane

Once done, patch update of the cluster resumes automatically.

Artifacts¶

This section lists the artifacts of components included in the Container Cloud patch release 2.27.4. For artifacts of the Cluster releases introduced in 2.27.4, see patch Cluster releases 16.2.4 and 17.2.4.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries ^Updated	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20240821131059
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20240821131059
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.40.23.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.40.23.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.40.23.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.40.23.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.40.23.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.40.23.tgz
Docker images	ambasador ^Updated	mirantis.azurecr.io/core/external/nginx:1.40.23
	baremetal-dnsmasq	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-27-alpine-20240806125028
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-2-27-alpine-20240827132225
	bm-collective	mirantis.azurecr.io/bm/bm-collective:base-2-27-alpine-20240812135414
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.40.23
	ironic	mirantis.azurecr.io/openstack/ironic:antelope-jammy-20240716113922
	ironic-inspector	mirantis.azurecr.io/openstack/ironic-inspector:antelope-jammy-20240716113922
	ironic-prometheus-exporter ^Updated	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20240819102310
	kaas-ipam	mirantis.azurecr.io/bm/kaas-ipam:base-2-27-alpine-20240812140336
	kubernetes-entrypoint ^Updated	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-4e381cb-20240813170642
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.17-focal-20240523075821
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.25.0-42-g8710cbe
	metallb-controller	mirantis.azurecr.io/bm/metallb/controller:v0.14.5-dfbd1a68-amd64
	metallb-speaker	mirantis.azurecr.io/bm/metallb/speaker:v0.14.5-dfbd1a68-amd64
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20240806124545

Core artifacts¶

Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.40.23.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.40.23.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.40.23.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.40.23.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.40.23.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.40.23.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.40.23.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.40.23.tgz
	host-os-modules-controller	https://binary.mirantis.com/core/helm/host-os-modules-controller-1.40.23.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.40.23.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.40.23.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.40.23.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.40.23.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.40.23.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.40.23.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.40.23.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.40.23.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.40.23.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.40.23.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.40.23.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.40.23.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.40.23.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.40.23.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.40.23.tgz
	secret-controller	https://binary.mirantis.com/core/helm/secret-controller-1.40.23.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.40.23.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.40.23
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.40.23
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.40.23
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.40.23
	cert-manager-controller ^Updated	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-7
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.40.23
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.40.23
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.40.23
	host-os-modules-controller ^Updated	mirantis.azurecr.io/core/host-os-modules-controller:1.40.23
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.40.23
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.40.23
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.40.23
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.40.23
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.40.23
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.40.23
	mcc-cache-warmup ^Updated	mirantis.azurecr.io/core/mcc-cache-warmup:1.40.23
	mcc-haproxy	mirantis.azurecr.io/lcm/mcc-haproxy:v0.25.0-42-g8710cbe
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.25.0-42-g8710cbe
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.40.23
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.40.23
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.40.23
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.40.23
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.40.23
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-11
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.40.23
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.40.23
	secret-controller ^Updated	mirantis.azurecr.io/core/secret-controller:1.40.23
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.40.23

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts ^Updated	iam	https://binary.mirantis.com/core/helm/iam-1.40.23.tgz
Docker images	kubectl	mirantis.azurecr.io/general/kubectl:20240711152257
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.17-focal-20240523075821
	mcc-keycloak	mirantis.azurecr.io/iam/mcc-keycloak:24.0.5-20240802071408

2.27.3¶

Important

For MOSK clusters, Container Cloud 2.27.3 is the first patch release of MOSK 24.2.x series using the patch Cluster release 17.2.3. For the update path of 24.1 and 24.2 series, see MOSK documentation: Cluster update scheme.

The Container Cloud patch release 2.27.3, which is based on the 2.27.0 major release, provides the following updates:

Support for the patch Cluster releases 16.2.3 and 17.2.3 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 24.2.1.
MKE:
- Support for MKE 3.7.12.
- Improvements in the MKE benchmark compliance (control ID 5.1.5): analyzed and fixed the majority of failed compliance checks for the following components:
  - Container Cloud: iam-keycloak in the kaas namespace and opensearch-dashboards in the stacklight namespace
  - MOSK: opensearch-dashboards in the stacklight namespace
Bare metal: update of Ubuntu mirror from ubuntu-2024-07-16-014744 to ubuntu-2024-08-06-014502 along with update of the minor kernel version from 5.15.0-116-generic to 5.15.0-117-generic.
VMware vSphere: suspension of support for cluster deployment, update, and attachment. For details, see Deprecation notes.
Security fixes for CVEs in images.

For main deliverables of the parent Container Cloud release of 2.27.3, refer to 2.27.0.

Security notes¶

In total, since Container Cloud 2.27.2, 1559 Common Vulnerabilities and Exposures (CVE) have been fixed in 2.27.3: 253 of critical and 1306 of high severity.

The table below includes the total numbers of addressed unique and common CVEs in images by product component since Container Cloud 2.27.2. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	Critical	High	Total
Ceph	Unique	3	14	17
Ceph	Common	142	736	878
Kaas core	Unique	4	22	26
Kaas core	Common	99	448	547
StackLight	Unique	7	51	58
StackLight	Common	12	122	134

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 24.2.1: Security notes.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.27.3 including the Cluster releases 16.2.3 and 17.2.3.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[47202] Inspection error on bare metal hosts after dnsmasq restart¶

Note

Moving forward, the workaround for this issue will be moved from Release Notes to MOSK Troubleshooting Guide: Inspection error on bare metal hosts after dnsmasq restart.

The issue can occur when:

The dnsmasq pod was moved to another node.
DHCP subnets were changed, including addition or removal. In this case, the dhcpd container of the dnsmasq pod is restarted.

Caution

If changing or adding of DHCP subnets is required to bootstrap new nodes, wait after changing or adding DHCP subnets until the dnsmasq pod becomes ready, then create BareMetalHost objects.

To verify whether the nodes are affected:

Verify whether the BareMetalHost objects contain the inspection error:

kubectl get bmh -n <managed-cluster-namespace-name>

Example of system response:

NAME            STATE         CONSUMER        ONLINE   ERROR              AGE
test-master-1   provisioned   test-master-1   true                        9d
test-master-2   provisioned   test-master-2   true                        9d
test-master-3   provisioned   test-master-3   true                        9d
test-worker-1   provisioned   test-worker-1   true                        9d
test-worker-2   provisioned   test-worker-2   true                        9d
test-worker-3   inspecting                    true     inspection error   19h

Verify whether the dnsmasq pod was in Ready state when the inspection of the affected baremetal hosts (test-worker-3 in the example above) was started:

kubectl -n kaas get pod <dnsmasq-pod-name> -oyaml

Example of system response:

...
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-10-10T15:37:34Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2024-10-11T07:38:54Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2024-10-11T07:38:54Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2024-10-10T15:37:34Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://6dbcf2fc4b36ce4c549c9191ab01f72d0236c51d42947675302675e4bfaf4cdf
    image: docker-dev-kaas-virtual.artifactory-eu.mcp.mirantis.net/bm/baremetal-dnsmasq:base-2-28-alpine-20240812132650
    imageID: docker-dev-kaas-virtual.artifactory-eu.mcp.mirantis.net/bm/baremetal-dnsmasq@sha256:3dad3e278add18e69b2608e462691c4823942641a0f0e25e6811e703e3c23b3b
    lastState:
      terminated:
        containerID: containerd://816fcf079cd544acd74e312065de5b5ed4dbf1dc6159fefffff4f644b5e45987
        exitCode: 0
        finishedAt: "2024-10-11T07:38:35Z"
        reason: Completed
        startedAt: "2024-10-10T15:37:45Z"
    name: dhcpd
    ready: true
    restartCount: 2
    started: true
    state:
      running:
        startedAt: "2024-10-11T07:38:37Z"
  ...

In the system response above, the dhcpd container was not ready between "2024-10-11T07:38:35Z" and "2024-10-11T07:38:54Z".

Verify the affected baremetal host. For example:

kubectl get bmh -n managed-ns test-worker-3 -oyaml

Example of system response:

...
status:
  errorCount: 15
  errorMessage: Introspection timeout
  errorType: inspection error
  ...
  operationHistory:
    deprovision:
      end: null
      start: null
    inspect:
      end: null
      start: "2024-10-11T07:38:19Z"
    provision:
      end: null
      start: null
    register:
      end: "2024-10-11T07:38:19Z"
      start: "2024-10-11T07:37:25Z"

Workaround

Reboot the node using the IPMI reset or cycle command.
If the node fails to boot, remove the failed BareMetalHost object and create it again:
1. Remove BareMetalHost object. For example:
```
kubectl delete bmh -n managed-ns test-worker-3
```
2. Verify that the BareMetalHost object is removed:
```
kubectl get bmh -n managed-ns test-worker-3
```
3. Create a BareMetalHost object from the template. For example:
```
kubectl create -f bmhc-test-worker-3.yaml
kubectl create -f bmh-test-worker-3.yaml
```

[46245] Lack of access permissions for HOC and HOCM objects¶

Fixed in 2.28.0 (17.3.0 and 16.3.0)

kubectl --kubeconfig ~/.kube/mgmt-config get hocm

Error from server (Forbidden): hostosconfigurationmodules.kaas.mirantis.com is forbidden:
User "2d74348b-5669-4c65-af31-6c05dbedac5f" cannot list resource "hostosconfigurationmodules"
in API group "kaas.mirantis.com" at the cluster scope: access denied

Workaround:

Modify the global-admin role by adding a new entry with the following contents to the rules list:

kubectl edit clusterroles kaas-global-admin

- apiGroups: [kaas.mirantis.com]
  resources: [hostosconfigurationmodules]
  verbs: ['*']

For each Container Cloud project, modify the kaas-operator role by adding a new entry with the following contents to the rules list:

kubectl -n <projectName> edit roles kaas-operator

- apiGroups: [kaas.mirantis.com]
  resources: [hostosconfigurations]
  verbs: ['*']

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[41305] DHCP responses are lost between dnsmasq and dhcp-relay pods¶

Fixed in 2.28.0 (17.3.0 and 16.3.0)

The issue was observed on environments having management cluster nodes configured with a single L2 segment used for all network traffic (PXE and LCM/management networks).

To verify whether the cluster is affected:

Verify whether the dnsmasq and dhcp-relay pods run on the same node in the management cluster:

kubectl -n kaas get pods -o wide| grep -e "dhcp\|dnsmasq"

Example of system response:

dhcp-relay-7d85f75f76-5vdw2   2/2   Running   2 (36h ago)   36h   10.10.0.122     kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>
dnsmasq-8f4b484b4-slhbd       5/5   Running   1 (36h ago)   36h   10.233.123.75   kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>

If this is the case, proceed to the workaround below.

Workaround:

Make sure that at least two management cluster nodes are schedulable:

kubectl get node

Example of a positive system response:

NAME                                             STATUS   ROLES    AGE   VERSION
kaas-node-bcedb87b-b3ce-46a4-a4ca-ea3068689e40   Ready    master   37h   v1.27.10-mirantis-1
kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   Ready    master   37h   v1.27.10-mirantis-1
kaas-node-ad5a6f51-b98f-43c3-91d5-55fed3d0ff21   Ready    master   37h   v1.27.10-mirantis-1

Delete the dhcp-relay pod:

kubectl -n kaas delete pod <dhcp-relay-xxxxx>

Verify that the dnsmasq and dhcp-relay pods are scheduled into different nodes:

kubectl -n kaas get pods -o wide| grep -e "dhcp\|dnsmasq"

Example of a positive system response:

dhcp-relay-7d85f75f76-rkv03   2/2   Running   0             49s   10.10.0.121     kaas-node-bcedb87b-b3ce-46a4-a4ca-ea3068689e40   <none>   <none>
dnsmasq-8f4b484b4-slhbd       5/5   Running   1 (37h ago)   37h   10.233.123.75   kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

LCM¶

[39437] Failure to replace a master node on a Container Cloud cluster¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

During the replacement of a master node on a cluster of any type, the process may get stuck with Kubelet's NodeReady condition is Unknown in the machine status on the remaining master nodes.

As a workaround, log in on the affected node and run the following command:

docker restart ucp-kubelet

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

[30294] Replacement of a master node is stuck on the calico-node Pod start¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a master node on a cluster of any type, the calico-node Pod fails to start on a new node that has the same IP address as the node being replaced.

Workaround:

From a CLI with an MKE client bundle, create a shell alias to start calicoctl using the mirantis/ucp-dsinfo image:

Since MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/key.pem \
-e ETCD_CA_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/ca.pem \
-e ETCD_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/cert.pem \
-v /var/run/calico:/var/run/calico \
-v /var/lib/docker/volumes/ucp-kv-certs/_data:/var/lib/docker/volumes/ucp-kv-certs/_data:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl \
"

Before MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
-e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
-e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
-v /var/run/calico:/var/run/calico \
-v ucp-node-certs:/ucp-node-certs:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl --allow-version-mismatch \
"

In the above command, replace the following values with the corresponding settings of the affected cluster:

<etcdEndpoint> is the etcd endpoint defined in the Calico configuration file. For example, ETCD_ENDPOINTS=127.0.0.1:12378
<mkeVersion> is the MKE version installed on your cluster. For example, mirantis/ucp-dsinfo:3.5.7.

Verify the node list on the cluster:
```
kubectl get node
```
Compare this list with the node list in Calico to identify the old node:
```
calicoctl get node -o wide
```

Remove the old node from Calico:

calicoctl delete node kaas-node-<nodeID>

[5782] Manager machine fails to be deployed during node replacement¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a manager machine, the following problems may occur:

The system adds the node to Docker swarm but not to Kubernetes
The node Deployment gets stuck with failed RethinkDB health checks

Workaround:

Delete the failed node.
Wait for the MKE cluster to become healthy. To monitor the cluster status:
1. Log in to the MKE web UI as described in Connect to the Mirantis Kubernetes Engine web UI.
2. Monitor the cluster status as described in MKE Operations Guide: Monitor an MKE cluster with the MKE web UI.
Deploy a new node.

[5568] The calico-kube-controllers Pod fails to clean up resources¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During the unsafe or forced deletion of a manager machine running the calico-kube-controllers Pod in the kube-system namespace, the following issues occur:

The calico-kube-controllers Pod fails to clean up resources associated with the deleted node
The calico-node Pod may fail to start up on a newly created node if the machine is provisioned with the same IP address as the deleted machine had

As a workaround, before deletion of the node running the calico-kube-controllers Pod, cordon and drain the node:

kubectl cordon <nodeName>
kubectl drain <nodeName>

Ceph¶

[50566] Ceph upgrade is very slow during patch or major cluster update¶

Fixed in 2.29.3 (17.3.8, 16.3.8, and 16.4.3)

 Warning  Unhealthy  57s (x16 over 3m27s)  kubelet  Startup probe failed:
 ceph daemon health check failed with the following output:
> no valid command found; 10 closest matches:
> 0
> 1
> 2
> abort
> assert
> bluefs debug_inject_read_zeros
> bluefs files list
> bluefs stats
> bluestore bluefs device info [<alloc_size:int>]
> config diff
> admin_socket: invalid command

Workaround:

Complete the following steps during every patch or major cluster update of the Cluster releases 17.2.x, 17.3.x, and 17.4.x (until Ceph 18.2.5 becomes supported):

Plan extra time in the maintenance window for the patch cluster update.

Slow starts will still impact the update procedure, but after completing the following step, the recovery process noticeably shortens without affecting the overall cluster state and data responsiveness.
Select one of the following options:
- Before the cluster update, set the noout flag:
```
ceph osd set noout
```
  Once the Ceph OSDs image upgrade is done, unset the flag:
```
ceph osd unset noout
```
- Monitor the Ceph OSDs image upgrade. If the symptoms of slow start appear, set the noout flag as soon as possible. Once the Ceph OSDs image upgrade is done, unset the flag.

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

Container Cloud web UI¶

[50181] Failure to deploy a compact cluster¶

To work around the issue, manually add the required labels using CLI. Once done, the cluster deployment resumes.

[50168] Inability to use a new project right after creation¶

A newly created project does not display all available tabs in the Container Cloud web UI and contains different access denied errors during first five minutes after creation.

To work around the issue, refresh the browser in five minutes after the project creation.

Artifacts¶

This section lists the artifacts of components included in the Container Cloud patch release 2.27.3. For artifacts of the Cluster releases introduced in 2.27.3, see patch Cluster releases 16.2.3 and 17.2.3.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20240716085444
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20240716085444
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.40.21.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.40.21.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.40.21.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.40.21.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.40.21.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.40.21.tgz
Docker images	ambasador ^Updated	mirantis.azurecr.io/core/external/nginx:1.40.21
	baremetal-dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-27-alpine-20240806125028
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-2-27-alpine-20240812133205
	bm-collective ^Updated	mirantis.azurecr.io/bm/bm-collective:base-2-27-alpine-20240812135414
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.40.21
	ironic	mirantis.azurecr.io/openstack/ironic:antelope-jammy-20240716113922
	ironic-inspector	mirantis.azurecr.io/openstack/ironic-inspector:antelope-jammy-20240716113922
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20240117102150
	kaas-ipam ^Updated	mirantis.azurecr.io/bm/kaas-ipam:base-2-27-alpine-20240812140336
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-ba8ada4-20240405150338
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.17-focal-20240523075821
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.25.0-42-g8710cbe
	metallb-controller ^Updated	mirantis.azurecr.io/bm/metallb/controller:v0.14.5-dfbd1a68-amd64
	metallb-speaker ^Updated	mirantis.azurecr.io/bm/metallb/speaker:v0.14.5-dfbd1a68-amd64
	syslog-ng ^Updated	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20240806124545

Core artifacts¶

Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.40.21.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.40.21.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.40.21.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.40.21.tgz
	byo-provider ^Unsupported	https://binary.mirantis.com/core/helm/byo-provider-1.40.21.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.40.21.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.40.21.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.40.21.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.40.21.tgz
	host-os-modules-controller	https://binary.mirantis.com/core/helm/host-os-modules-controller-1.40.21.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.40.21.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.40.21.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.40.21.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.40.21.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.40.21.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.40.21.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.40.21.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.40.21.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.40.21.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.40.21.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.40.21.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.40.21.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.40.21.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.40.21.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.40.21.tgz
	secret-controller	https://binary.mirantis.com/core/helm/secret-controller-1.40.21.tgz
	squid-proxy ^Unsupported	https://binary.mirantis.com/core/helm/squid-proxy-1.40.21.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.40.21.tgz
	vsphere-credentials-controller ^Unsupported	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.40.21.tgz
	vsphere-provider ^Unsupported	https://binary.mirantis.com/core/helm/vsphere-provider-1.40.21.tgz
	vsphere-vm-template-controller ^Unsupported	https://binary.mirantis.com/core/helm/vsphere-vm-template-controller-1.40.21.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.40.21
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.40.21
	byo-cluster-api-controller ^Unsupported	mirantis.azurecr.io/core/byo-cluster-api-controller:1.40.21
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.40.21
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-6
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.40.21
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.40.21
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.40.21
	host-os-modules-controller ^Updated	mirantis.azurecr.io/core/host-os-modules-controller:1.40.21
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.40.21
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.40.21
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.40.21
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.40.21
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.40.21
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.40.21
	mcc-cache-warmup ^Updated	mirantis.azurecr.io/core/mcc-cache-warmup:1.40.21
	mcc-haproxy ^Updated	mirantis.azurecr.io/lcm/mcc-haproxy:v0.25.0-42-g8710cbe
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.25.0-42-g8710cbe
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.40.21
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.40.21
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.40.21
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.40.21
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.40.21
	registry ^Updated	mirantis.azurecr.io/lcm/registry:v2.8.1-11
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.40.21
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.40.21
	secret-controller ^Updated	mirantis.azurecr.io/core/secret-controller:1.40.21
	squid-proxy ^Unsupported	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-10-g24a0d69
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.40.21
	vsphere-cluster-api-controller ^Unsupported	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.40.21
	vsphere-credentials-controller ^Unsupported	mirantis.azurecr.io/core/vsphere-credentials-controller:1.40.21
	vsphere-vm-template-controller ^Unsupported	mirantis.azurecr.io/core/vsphere-vm-template-controller:1.40.21

IAM artifacts¶

IAM artifacts¶
Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts ^Updated	iam	https://binary.mirantis.com/core/helm/iam-1.40.21.tgz
Docker images	kubectl	mirantis.azurecr.io/general/kubectl:20240711152257
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.17-focal-20240523075821
	mcc-keycloak ^Updated	mirantis.azurecr.io/iam/mcc-keycloak:24.0.5-20240802071408

2.27.2¶

Important

For MOSK clusters, Container Cloud 2.27.2 is the continuation for MOSK 24.1.x series using the patch Cluster release 17.1.7. For the update path of 24.1 and 24.2 series, see MOSK documentation: Cluster update scheme.

The management cluster of a MOSK 24.1, 24.1.5, or 24.1.6 cluster is automatically updated to the latest patch Cluster release 16.2.2.

The Container Cloud patch release 2.27.2, which is based on the 2.27.0 major release, provides the following updates:

Support for the patch Cluster release 16.2.2.
Support for the patch Cluster releases 16.1.7 and 17.1.7 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 24.1.7.
Support for MKE 3.7.11.
Bare metal: update of Ubuntu mirror from ubuntu-2024-06-27-095142 to ubuntu-2024-07-16-014744 along with update of minor kernel version from 5.15.0-113-generic to 5.15.0-116-generic (Cluster release 16.2.2).
Security fixes for CVEs in images.

For main deliverables of the parent Container Cloud release of 2.27.2, refer to 2.27.0.

Security notes¶

In total, since Container Cloud 2.27.1, 95 Common Vulnerabilities and Exposures (CVE) have been fixed in 2.27.2: 6 of critical and 89 of high severity.

The table below includes the total numbers of addressed unique and common CVEs in images by product component since Container Cloud 2.27.1. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	Critical	High	Total
Kaas core	Unique	5	26	31
Kaas core	Common	6	69	75
StackLight	Unique	0	3	3
StackLight	Common	0	20	20

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 24.1.7: Security notes.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.27.2 including the Cluster releases 16.2.2, 16.1.7, and 17.1.7.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[47202] Inspection error on bare metal hosts after dnsmasq restart¶

Note

Moving forward, the workaround for this issue will be moved from Release Notes to MOSK Troubleshooting Guide: Inspection error on bare metal hosts after dnsmasq restart.

The issue can occur when:

The dnsmasq pod was moved to another node.
DHCP subnets were changed, including addition or removal. In this case, the dhcpd container of the dnsmasq pod is restarted.

Caution

If changing or adding of DHCP subnets is required to bootstrap new nodes, wait after changing or adding DHCP subnets until the dnsmasq pod becomes ready, then create BareMetalHost objects.

To verify whether the nodes are affected:

Verify whether the BareMetalHost objects contain the inspection error:

kubectl get bmh -n <managed-cluster-namespace-name>

Example of system response:

NAME            STATE         CONSUMER        ONLINE   ERROR              AGE
test-master-1   provisioned   test-master-1   true                        9d
test-master-2   provisioned   test-master-2   true                        9d
test-master-3   provisioned   test-master-3   true                        9d
test-worker-1   provisioned   test-worker-1   true                        9d
test-worker-2   provisioned   test-worker-2   true                        9d
test-worker-3   inspecting                    true     inspection error   19h

Verify whether the dnsmasq pod was in Ready state when the inspection of the affected baremetal hosts (test-worker-3 in the example above) was started:

kubectl -n kaas get pod <dnsmasq-pod-name> -oyaml

Example of system response:

...
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-10-10T15:37:34Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2024-10-11T07:38:54Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2024-10-11T07:38:54Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2024-10-10T15:37:34Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://6dbcf2fc4b36ce4c549c9191ab01f72d0236c51d42947675302675e4bfaf4cdf
    image: docker-dev-kaas-virtual.artifactory-eu.mcp.mirantis.net/bm/baremetal-dnsmasq:base-2-28-alpine-20240812132650
    imageID: docker-dev-kaas-virtual.artifactory-eu.mcp.mirantis.net/bm/baremetal-dnsmasq@sha256:3dad3e278add18e69b2608e462691c4823942641a0f0e25e6811e703e3c23b3b
    lastState:
      terminated:
        containerID: containerd://816fcf079cd544acd74e312065de5b5ed4dbf1dc6159fefffff4f644b5e45987
        exitCode: 0
        finishedAt: "2024-10-11T07:38:35Z"
        reason: Completed
        startedAt: "2024-10-10T15:37:45Z"
    name: dhcpd
    ready: true
    restartCount: 2
    started: true
    state:
      running:
        startedAt: "2024-10-11T07:38:37Z"
  ...

In the system response above, the dhcpd container was not ready between "2024-10-11T07:38:35Z" and "2024-10-11T07:38:54Z".

Verify the affected baremetal host. For example:

kubectl get bmh -n managed-ns test-worker-3 -oyaml

Example of system response:

...
status:
  errorCount: 15
  errorMessage: Introspection timeout
  errorType: inspection error
  ...
  operationHistory:
    deprovision:
      end: null
      start: null
    inspect:
      end: null
      start: "2024-10-11T07:38:19Z"
    provision:
      end: null
      start: null
    register:
      end: "2024-10-11T07:38:19Z"
      start: "2024-10-11T07:37:25Z"

Workaround

Reboot the node using the IPMI reset or cycle command.
If the node fails to boot, remove the failed BareMetalHost object and create it again:
1. Remove BareMetalHost object. For example:
```
kubectl delete bmh -n managed-ns test-worker-3
```
2. Verify that the BareMetalHost object is removed:
```
kubectl get bmh -n managed-ns test-worker-3
```
3. Create a BareMetalHost object from the template. For example:
```
kubectl create -f bmhc-test-worker-3.yaml
kubectl create -f bmh-test-worker-3.yaml
```

[46245] Lack of access permissions for HOC and HOCM objects¶

Fixed in 2.28.0 (17.3.0 and 16.3.0)

kubectl --kubeconfig ~/.kube/mgmt-config get hocm

Error from server (Forbidden): hostosconfigurationmodules.kaas.mirantis.com is forbidden:
User "2d74348b-5669-4c65-af31-6c05dbedac5f" cannot list resource "hostosconfigurationmodules"
in API group "kaas.mirantis.com" at the cluster scope: access denied

Workaround:

Modify the global-admin role by adding a new entry with the following contents to the rules list:

kubectl edit clusterroles kaas-global-admin

- apiGroups: [kaas.mirantis.com]
  resources: [hostosconfigurationmodules]
  verbs: ['*']

For each Container Cloud project, modify the kaas-operator role by adding a new entry with the following contents to the rules list:

kubectl -n <projectName> edit roles kaas-operator

- apiGroups: [kaas.mirantis.com]
  resources: [hostosconfigurations]
  verbs: ['*']

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[41305] DHCP responses are lost between dnsmasq and dhcp-relay pods¶

Fixed in 2.28.0 (17.3.0 and 16.3.0)

The issue was observed on environments having management cluster nodes configured with a single L2 segment used for all network traffic (PXE and LCM/management networks).

To verify whether the cluster is affected:

Verify whether the dnsmasq and dhcp-relay pods run on the same node in the management cluster:

kubectl -n kaas get pods -o wide| grep -e "dhcp\|dnsmasq"

Example of system response:

dhcp-relay-7d85f75f76-5vdw2   2/2   Running   2 (36h ago)   36h   10.10.0.122     kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>
dnsmasq-8f4b484b4-slhbd       5/5   Running   1 (36h ago)   36h   10.233.123.75   kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>

If this is the case, proceed to the workaround below.

Workaround:

Make sure that at least two management cluster nodes are schedulable:

kubectl get node

Example of a positive system response:

NAME                                             STATUS   ROLES    AGE   VERSION
kaas-node-bcedb87b-b3ce-46a4-a4ca-ea3068689e40   Ready    master   37h   v1.27.10-mirantis-1
kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   Ready    master   37h   v1.27.10-mirantis-1
kaas-node-ad5a6f51-b98f-43c3-91d5-55fed3d0ff21   Ready    master   37h   v1.27.10-mirantis-1

Delete the dhcp-relay pod:

kubectl -n kaas delete pod <dhcp-relay-xxxxx>

Verify that the dnsmasq and dhcp-relay pods are scheduled into different nodes:

kubectl -n kaas get pods -o wide| grep -e "dhcp\|dnsmasq"

Example of a positive system response:

dhcp-relay-7d85f75f76-rkv03   2/2   Running   0             49s   10.10.0.121     kaas-node-bcedb87b-b3ce-46a4-a4ca-ea3068689e40   <none>   <none>
dnsmasq-8f4b484b4-slhbd       5/5   Running   1 (37h ago)   37h   10.233.123.75   kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

LCM¶

[39437] Failure to replace a master node on a Container Cloud cluster¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

During the replacement of a master node on a cluster of any type, the process may get stuck with Kubelet's NodeReady condition is Unknown in the machine status on the remaining master nodes.

As a workaround, log in on the affected node and run the following command:

docker restart ucp-kubelet

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

[30294] Replacement of a master node is stuck on the calico-node Pod start¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a master node on a cluster of any type, the calico-node Pod fails to start on a new node that has the same IP address as the node being replaced.

Workaround:

From a CLI with an MKE client bundle, create a shell alias to start calicoctl using the mirantis/ucp-dsinfo image:

Since MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/key.pem \
-e ETCD_CA_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/ca.pem \
-e ETCD_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/cert.pem \
-v /var/run/calico:/var/run/calico \
-v /var/lib/docker/volumes/ucp-kv-certs/_data:/var/lib/docker/volumes/ucp-kv-certs/_data:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl \
"

Before MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
-e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
-e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
-v /var/run/calico:/var/run/calico \
-v ucp-node-certs:/ucp-node-certs:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl --allow-version-mismatch \
"

In the above command, replace the following values with the corresponding settings of the affected cluster:

<etcdEndpoint> is the etcd endpoint defined in the Calico configuration file. For example, ETCD_ENDPOINTS=127.0.0.1:12378
<mkeVersion> is the MKE version installed on your cluster. For example, mirantis/ucp-dsinfo:3.5.7.

Verify the node list on the cluster:
```
kubectl get node
```
Compare this list with the node list in Calico to identify the old node:
```
calicoctl get node -o wide
```

Remove the old node from Calico:

calicoctl delete node kaas-node-<nodeID>

[5782] Manager machine fails to be deployed during node replacement¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a manager machine, the following problems may occur:

The system adds the node to Docker swarm but not to Kubernetes
The node Deployment gets stuck with failed RethinkDB health checks

Workaround:

Delete the failed node.
Wait for the MKE cluster to become healthy. To monitor the cluster status:
1. Log in to the MKE web UI as described in Connect to the Mirantis Kubernetes Engine web UI.
2. Monitor the cluster status as described in MKE Operations Guide: Monitor an MKE cluster with the MKE web UI.
Deploy a new node.

[5568] The calico-kube-controllers Pod fails to clean up resources¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During the unsafe or forced deletion of a manager machine running the calico-kube-controllers Pod in the kube-system namespace, the following issues occur:

The calico-kube-controllers Pod fails to clean up resources associated with the deleted node
The calico-node Pod may fail to start up on a newly created node if the machine is provisioned with the same IP address as the deleted machine had

As a workaround, before deletion of the node running the calico-kube-controllers Pod, cordon and drain the node:

kubectl cordon <nodeName>
kubectl drain <nodeName>

Ceph¶

[50566] Ceph upgrade is very slow during patch or major cluster update¶

Fixed in 2.29.3 (17.3.8, 16.3.8, and 16.4.3)

 Warning  Unhealthy  57s (x16 over 3m27s)  kubelet  Startup probe failed:
 ceph daemon health check failed with the following output:
> no valid command found; 10 closest matches:
> 0
> 1
> 2
> abort
> assert
> bluefs debug_inject_read_zeros
> bluefs files list
> bluefs stats
> bluestore bluefs device info [<alloc_size:int>]
> config diff
> admin_socket: invalid command

Workaround:

Complete the following steps during every patch or major cluster update of the Cluster releases 17.2.x, 17.3.x, and 17.4.x (until Ceph 18.2.5 becomes supported):

Plan extra time in the maintenance window for the patch cluster update.

Slow starts will still impact the update procedure, but after completing the following step, the recovery process noticeably shortens without affecting the overall cluster state and data responsiveness.
Select one of the following options:
- Before the cluster update, set the noout flag:
```
ceph osd set noout
```
  Once the Ceph OSDs image upgrade is done, unset the flag:
```
ceph osd unset noout
```
- Monitor the Ceph OSDs image upgrade. If the symptoms of slow start appear, set the noout flag as soon as possible. Once the Ceph OSDs image upgrade is done, unset the flag.

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

Container Cloud web UI¶

[50181] Failure to deploy a compact cluster¶

To work around the issue, manually add the required labels using CLI. Once done, the cluster deployment resumes.

[50168] Inability to use a new project right after creation¶

A newly created project does not display all available tabs in the Container Cloud web UI and contains different access denied errors during first five minutes after creation.

To work around the issue, refresh the browser in five minutes after the project creation.

Artifacts¶

This section lists the artifacts of components included in the Container Cloud patch release 2.27.2. For artifacts of the Cluster releases introduced in 2.27.2, see patch Cluster releases 16.2.2, 16.1.7, and 17.1.7.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries ^Updated	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20240716085444
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20240716085444
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.40.18.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.40.18.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.40.18.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.40.18.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.40.18.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.40.18.tgz
Docker images	ambasador ^Updated	mirantis.azurecr.io/core/external/nginx:1.40.18
	baremetal-dnsmasq	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-27-alpine-20240701130209
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-2-27-alpine-20240711081559
	bm-collective	mirantis.azurecr.io/bm/bm-collective:base-2-27-alpine-20240701130719
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.40.18
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:antelope-jammy-20240716113922
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:antelope-jammy-20240716113922
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20240117102150
	kaas-ipam	mirantis.azurecr.io/bm/kaas-ipam:base-2-27-alpine-20240701133222
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-ba8ada4-20240405150338
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.17-focal-20240523075821
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.25.0-40-g890ffca
	metallb-controller	mirantis.azurecr.io/bm/metallb/controller:v0.14.5-e86184d9-amd64
	metallb-speaker	mirantis.azurecr.io/bm/metallb/speaker:v0.14.5-e86184d9-amd64
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20240701125905

Core artifacts¶

Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.40.18.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.40.18.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.40.18.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.40.18.tgz
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.40.18.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.40.18.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.40.18.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.40.18.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.40.18.tgz
	host-os-modules-controller	https://binary.mirantis.com/core/helm/host-os-modules-controller-1.40.18.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.40.18.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.40.18.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.40.18.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.40.18.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.40.18.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.40.18.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.40.18.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.40.18.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.40.18.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.40.18.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.40.18.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.40.18.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.40.18.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.40.18.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.40.18.tgz
	secret-controller	https://binary.mirantis.com/core/helm/secret-controller-1.40.18.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.40.18.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.40.18.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.40.18.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.40.18.tgz
	vsphere-vm-template-controller	https://binary.mirantis.com/core/helm/vsphere-vm-template-controller-1.40.18.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.40.18
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.40.18
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.40.18
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.40.18
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-6
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.40.18
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.40.18
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.40.18
	host-os-modules-controller ^Updated	mirantis.azurecr.io/core/host-os-modules-controller:1.40.18
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.40.18
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.40.18
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.40.18
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.40.18
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.40.18
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.40.18
	mcc-haproxy	mirantis.azurecr.io/lcm/mcc-haproxy:v0.25.0-40-g890ffca
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.25.0-40-g890ffca
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.40.18
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.40.18
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.40.18
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.40.18
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.40.18
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-10
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.40.18
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.40.18
	secret-controller ^Updated	mirantis.azurecr.io/core/secret-controller:1.40.18
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-10-g24a0d69
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.40.18
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.40.18
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.40.18
	vsphere-vm-template-controller ^Updated	mirantis.azurecr.io/core/vsphere-vm-template-controller:1.40.18

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts ^Updated	iam	https://binary.mirantis.com/core/helm/iam-1.40.18.tgz
Docker images	kubectl ^Updated	mirantis.azurecr.io/general/kubectl:20240711152257
	kubernetes-entrypoint ^Removed	n/a
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.17-focal-20240523075821
	mcc-keycloak	mirantis.azurecr.io/iam/mcc-keycloak:24.0.5-20240621131831

2.27.1¶

Important

For MOSK clusters, Container Cloud 2.27.1 is the continuation for MOSK 24.1.x series using the patch Cluster release 17.1.6. For the update path of 24.1 and 24.2 series, see MOSK documentation: Cluster update scheme.

The management cluster of a MOSK 24.1 or 24.1.5 cluster is automatically updated to the latest patch Cluster release 16.2.1.

The Container Cloud patch release 2.27.1, which is based on the 2.27.0 major release, provides the following updates:

Support for the patch Cluster release 16.2.1.
Support for the patch Cluster releases 16.1.6 and 17.1.6 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 24.1.6.
Support for MKE 3.7.10.
Support for docker-ee-cli 23.0.13 in MCR 23.0.11 to fix several CVEs.
Bare metal: update of Ubuntu mirror from ubuntu-2024-05-17-013445 to ubuntu-2024-06-27-095142 along with update of minor kernel version from 5.15.0-107-generic to 5.15.0-113-generic.
Security fixes for CVEs in images.
Bug fixes.

For main deliverables of the parent Container Cloud release of 2.27.1, refer to 2.27.0.

Security notes¶

In total, since Container Cloud 2.27.0, 270 Common Vulnerabilities and Exposures (CVE) of high severity have been fixed in 2.27.1.

The table below includes the total numbers of addressed unique and common CVEs in images by product component since Container Cloud 2.27.0. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	High	Total
Ceph	Unique	6	6
Ceph	Common	29	29
Kaas core	Unique	10	10
Kaas core	Common	178	178
StackLight	Unique	14	14
StackLight	Common	63	63

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 24.1.6: Security notes.

Addressed issues¶

The following issues have been addressed in the Container Cloud patch release 2.27.1 along with the patch Cluster releases 16.2.1, 16.1.6, and 17.1.6.

[42304] [StackLight] [Cluster releases 17.1.6, 16.1.6] Fixed the issue with failure of shard relocation in the OpenSearch cluster on large Container Cloud managed clusters.
[40020] [StackLight] [Cluster releases 17.1.6, 16.1.6] Fixed the issue with rollover_policy not being applied to the current indices while updating the policy for the current system* and audit* data streams.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.27.1 including the Cluster releases 16.2.1, 16.1.6, and 17.1.6.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[47202] Inspection error on bare metal hosts after dnsmasq restart¶

Note

Moving forward, the workaround for this issue will be moved from Release Notes to MOSK Troubleshooting Guide: Inspection error on bare metal hosts after dnsmasq restart.

The issue can occur when:

The dnsmasq pod was moved to another node.
DHCP subnets were changed, including addition or removal. In this case, the dhcpd container of the dnsmasq pod is restarted.

Caution

If changing or adding of DHCP subnets is required to bootstrap new nodes, wait after changing or adding DHCP subnets until the dnsmasq pod becomes ready, then create BareMetalHost objects.

To verify whether the nodes are affected:

Verify whether the BareMetalHost objects contain the inspection error:

kubectl get bmh -n <managed-cluster-namespace-name>

Example of system response:

NAME            STATE         CONSUMER        ONLINE   ERROR              AGE
test-master-1   provisioned   test-master-1   true                        9d
test-master-2   provisioned   test-master-2   true                        9d
test-master-3   provisioned   test-master-3   true                        9d
test-worker-1   provisioned   test-worker-1   true                        9d
test-worker-2   provisioned   test-worker-2   true                        9d
test-worker-3   inspecting                    true     inspection error   19h

Verify whether the dnsmasq pod was in Ready state when the inspection of the affected baremetal hosts (test-worker-3 in the example above) was started:

kubectl -n kaas get pod <dnsmasq-pod-name> -oyaml

Example of system response:

...
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-10-10T15:37:34Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2024-10-11T07:38:54Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2024-10-11T07:38:54Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2024-10-10T15:37:34Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://6dbcf2fc4b36ce4c549c9191ab01f72d0236c51d42947675302675e4bfaf4cdf
    image: docker-dev-kaas-virtual.artifactory-eu.mcp.mirantis.net/bm/baremetal-dnsmasq:base-2-28-alpine-20240812132650
    imageID: docker-dev-kaas-virtual.artifactory-eu.mcp.mirantis.net/bm/baremetal-dnsmasq@sha256:3dad3e278add18e69b2608e462691c4823942641a0f0e25e6811e703e3c23b3b
    lastState:
      terminated:
        containerID: containerd://816fcf079cd544acd74e312065de5b5ed4dbf1dc6159fefffff4f644b5e45987
        exitCode: 0
        finishedAt: "2024-10-11T07:38:35Z"
        reason: Completed
        startedAt: "2024-10-10T15:37:45Z"
    name: dhcpd
    ready: true
    restartCount: 2
    started: true
    state:
      running:
        startedAt: "2024-10-11T07:38:37Z"
  ...

In the system response above, the dhcpd container was not ready between "2024-10-11T07:38:35Z" and "2024-10-11T07:38:54Z".

Verify the affected baremetal host. For example:

kubectl get bmh -n managed-ns test-worker-3 -oyaml

Example of system response:

...
status:
  errorCount: 15
  errorMessage: Introspection timeout
  errorType: inspection error
  ...
  operationHistory:
    deprovision:
      end: null
      start: null
    inspect:
      end: null
      start: "2024-10-11T07:38:19Z"
    provision:
      end: null
      start: null
    register:
      end: "2024-10-11T07:38:19Z"
      start: "2024-10-11T07:37:25Z"

Workaround

Reboot the node using the IPMI reset or cycle command.
If the node fails to boot, remove the failed BareMetalHost object and create it again:
1. Remove BareMetalHost object. For example:
```
kubectl delete bmh -n managed-ns test-worker-3
```
2. Verify that the BareMetalHost object is removed:
```
kubectl get bmh -n managed-ns test-worker-3
```
3. Create a BareMetalHost object from the template. For example:
```
kubectl create -f bmhc-test-worker-3.yaml
kubectl create -f bmh-test-worker-3.yaml
```

[46245] Lack of access permissions for HOC and HOCM objects¶

Fixed in 2.28.0 (17.3.0 and 16.3.0)

kubectl --kubeconfig ~/.kube/mgmt-config get hocm

Error from server (Forbidden): hostosconfigurationmodules.kaas.mirantis.com is forbidden:
User "2d74348b-5669-4c65-af31-6c05dbedac5f" cannot list resource "hostosconfigurationmodules"
in API group "kaas.mirantis.com" at the cluster scope: access denied

Workaround:

Modify the global-admin role by adding a new entry with the following contents to the rules list:

kubectl edit clusterroles kaas-global-admin

- apiGroups: [kaas.mirantis.com]
  resources: [hostosconfigurationmodules]
  verbs: ['*']

For each Container Cloud project, modify the kaas-operator role by adding a new entry with the following contents to the rules list:

kubectl -n <projectName> edit roles kaas-operator

- apiGroups: [kaas.mirantis.com]
  resources: [hostosconfigurations]
  verbs: ['*']

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[41305] DHCP responses are lost between dnsmasq and dhcp-relay pods¶

Fixed in 2.28.0 (17.3.0 and 16.3.0)

The issue was observed on environments having management cluster nodes configured with a single L2 segment used for all network traffic (PXE and LCM/management networks).

To verify whether the cluster is affected:

Verify whether the dnsmasq and dhcp-relay pods run on the same node in the management cluster:

kubectl -n kaas get pods -o wide| grep -e "dhcp\|dnsmasq"

Example of system response:

dhcp-relay-7d85f75f76-5vdw2   2/2   Running   2 (36h ago)   36h   10.10.0.122     kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>
dnsmasq-8f4b484b4-slhbd       5/5   Running   1 (36h ago)   36h   10.233.123.75   kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>

If this is the case, proceed to the workaround below.

Workaround:

Make sure that at least two management cluster nodes are schedulable:

kubectl get node

Example of a positive system response:

NAME                                             STATUS   ROLES    AGE   VERSION
kaas-node-bcedb87b-b3ce-46a4-a4ca-ea3068689e40   Ready    master   37h   v1.27.10-mirantis-1
kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   Ready    master   37h   v1.27.10-mirantis-1
kaas-node-ad5a6f51-b98f-43c3-91d5-55fed3d0ff21   Ready    master   37h   v1.27.10-mirantis-1

Delete the dhcp-relay pod:

kubectl -n kaas delete pod <dhcp-relay-xxxxx>

Verify that the dnsmasq and dhcp-relay pods are scheduled into different nodes:

kubectl -n kaas get pods -o wide| grep -e "dhcp\|dnsmasq"

Example of a positive system response:

dhcp-relay-7d85f75f76-rkv03   2/2   Running   0             49s   10.10.0.121     kaas-node-bcedb87b-b3ce-46a4-a4ca-ea3068689e40   <none>   <none>
dnsmasq-8f4b484b4-slhbd       5/5   Running   1 (37h ago)   37h   10.233.123.75   kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

LCM¶

[39437] Failure to replace a master node on a Container Cloud cluster¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

During the replacement of a master node on a cluster of any type, the process may get stuck with Kubelet's NodeReady condition is Unknown in the machine status on the remaining master nodes.

As a workaround, log in on the affected node and run the following command:

docker restart ucp-kubelet

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

[30294] Replacement of a master node is stuck on the calico-node Pod start¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a master node on a cluster of any type, the calico-node Pod fails to start on a new node that has the same IP address as the node being replaced.

Workaround:

From a CLI with an MKE client bundle, create a shell alias to start calicoctl using the mirantis/ucp-dsinfo image:

Since MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/key.pem \
-e ETCD_CA_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/ca.pem \
-e ETCD_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/cert.pem \
-v /var/run/calico:/var/run/calico \
-v /var/lib/docker/volumes/ucp-kv-certs/_data:/var/lib/docker/volumes/ucp-kv-certs/_data:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl \
"

Before MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
-e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
-e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
-v /var/run/calico:/var/run/calico \
-v ucp-node-certs:/ucp-node-certs:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl --allow-version-mismatch \
"

In the above command, replace the following values with the corresponding settings of the affected cluster:

<etcdEndpoint> is the etcd endpoint defined in the Calico configuration file. For example, ETCD_ENDPOINTS=127.0.0.1:12378
<mkeVersion> is the MKE version installed on your cluster. For example, mirantis/ucp-dsinfo:3.5.7.

Verify the node list on the cluster:
```
kubectl get node
```
Compare this list with the node list in Calico to identify the old node:
```
calicoctl get node -o wide
```

Remove the old node from Calico:

calicoctl delete node kaas-node-<nodeID>

[5782] Manager machine fails to be deployed during node replacement¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a manager machine, the following problems may occur:

The system adds the node to Docker swarm but not to Kubernetes
The node Deployment gets stuck with failed RethinkDB health checks

Workaround:

Delete the failed node.
Wait for the MKE cluster to become healthy. To monitor the cluster status:
1. Log in to the MKE web UI as described in Connect to the Mirantis Kubernetes Engine web UI.
2. Monitor the cluster status as described in MKE Operations Guide: Monitor an MKE cluster with the MKE web UI.
Deploy a new node.

[5568] The calico-kube-controllers Pod fails to clean up resources¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During the unsafe or forced deletion of a manager machine running the calico-kube-controllers Pod in the kube-system namespace, the following issues occur:

The calico-kube-controllers Pod fails to clean up resources associated with the deleted node
The calico-node Pod may fail to start up on a newly created node if the machine is provisioned with the same IP address as the deleted machine had

As a workaround, before deletion of the node running the calico-kube-controllers Pod, cordon and drain the node:

kubectl cordon <nodeName>
kubectl drain <nodeName>

Ceph¶

[50566] Ceph upgrade is very slow during patch or major cluster update¶

Fixed in 2.29.3 (17.3.8, 16.3.8, and 16.4.3)

 Warning  Unhealthy  57s (x16 over 3m27s)  kubelet  Startup probe failed:
 ceph daemon health check failed with the following output:
> no valid command found; 10 closest matches:
> 0
> 1
> 2
> abort
> assert
> bluefs debug_inject_read_zeros
> bluefs files list
> bluefs stats
> bluestore bluefs device info [<alloc_size:int>]
> config diff
> admin_socket: invalid command

Workaround:

Complete the following steps during every patch or major cluster update of the Cluster releases 17.2.x, 17.3.x, and 17.4.x (until Ceph 18.2.5 becomes supported):

Plan extra time in the maintenance window for the patch cluster update.

Slow starts will still impact the update procedure, but after completing the following step, the recovery process noticeably shortens without affecting the overall cluster state and data responsiveness.
Select one of the following options:
- Before the cluster update, set the noout flag:
```
ceph osd set noout
```
  Once the Ceph OSDs image upgrade is done, unset the flag:
```
ceph osd unset noout
```
- Monitor the Ceph OSDs image upgrade. If the symptoms of slow start appear, set the noout flag as soon as possible. Once the Ceph OSDs image upgrade is done, unset the flag.

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

Container Cloud web UI¶

[50181] Failure to deploy a compact cluster¶

To work around the issue, manually add the required labels using CLI. Once done, the cluster deployment resumes.

[50168] Inability to use a new project right after creation¶

A newly created project does not display all available tabs in the Container Cloud web UI and contains different access denied errors during first five minutes after creation.

To work around the issue, refresh the browser in five minutes after the project creation.

Update notes¶

This section describes the specific actions you as a cloud operator need to complete before or after your Container Cloud cluster update to the Cluster releases 17.1.6, 16.2.1, or 16.1.6.

Consider this information as a supplement to the generic update procedures published in Operations Guide: Automatic upgrade of a management cluster and Update a patch Cluster release of a managed cluster.

Post-update actions¶

Prepare for changing label values in Ceph metrics used in customizations¶

Note

If you do not use Ceph metrics in any customizations, for example, custom alerts, Grafana dashboards, or queries in custom workloads, skip this section.

After deprecating the performance metric exporter that is integrated into the Ceph Manager daemon for the sake of the dedicated Ceph Exporter daemon in Container Cloud 2.27.0, you may need to prepare for updating values of several labels in Ceph metrics if you use them in any customizations such as custom alerts, Grafana dashboards, or queries in custom tools. These labels will be changed in Container Cloud 2.28.0 (Cluster releases 16.3.0 and 17.3.0).

Note

Names of metrics will not be changed, no metrics will be removed.

All Ceph metrics to be collected by the Ceph Exporter daemon will change their labels job and instance due to scraping metrics from new Ceph Exporter daemon instead of the performance metric exporter of Ceph Manager:

Values of the job labels will be changed from rook-ceph-mgr to prometheus-rook-exporter for all Ceph metrics moved to Ceph Exporter. The full list of moved metrics is presented below.
Values of the instance labels will be changed from the metric endpoint of Ceph Manager with port 9283 to the metric endpoint of Ceph Exporter with port 9926 for all Ceph metrics moved to Ceph Exporter. The full list of moved metrics is presented below.
Values of the instance_id labels of Ceph metrics from the RADOS Gateway (RGW) daemons will be changed from the daemon GID to the daemon subname. For example, instead of instance_id="<RGW_PROCESS_GID>", the instance_id="a" (ceph_rgw_qlen{instance_id="a"}) will be used. The list of moved Ceph RGW metrics is presented below.

Artifacts¶

This section lists the artifacts of components included in the Container Cloud patch release 2.27.1. For artifacts of the Cluster releases introduced in 2.27.1, see patch Cluster releases 16.2.1, 16.1.6, and 17.1.6.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries ^Updated	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20240627104414
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20240627104414
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.40.15.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.40.15.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.40.15.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.40.15.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.40.15.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.40.15.tgz
Docker images	ambasador ^Updated	mirantis.azurecr.io/core/external/nginx:1.40.15
	baremetal-dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-27-alpine-20240701130209
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-2-27-alpine-20240701130001
	bm-collective ^Updated	mirantis.azurecr.io/bm/bm-collective:base-2-27-alpine-20240701130719
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.40.15
	ironic	mirantis.azurecr.io/openstack/ironic:antelope-jammy-20240522120643
	ironic-inspector	mirantis.azurecr.io/openstack/ironic-inspector:antelope-jammy-20240522120643
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20240117102150
	kaas-ipam ^Updated	mirantis.azurecr.io/bm/kaas-ipam:base-2-27-alpine-20240701133222
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-ba8ada4-20240405150338
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.17-focal-20240523075821
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.25.0-40-g890ffca
	metallb-controller	mirantis.azurecr.io/bm/metallb/controller:v0.14.5-e86184d9-amd64
	metallb-speaker	mirantis.azurecr.io/bm/metallb/speaker:v0.14.5-e86184d9-amd64
	syslog-ng ^Updated	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20240701125905

Core artifacts¶

Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.40.15.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.40.15.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.40.15.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.40.15.tgz
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.40.15.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.40.15.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.40.15.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.40.15.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.40.15.tgz
	host-os-modules-controller	https://binary.mirantis.com/core/helm/host-os-modules-controller-1.40.15.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.40.15.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.40.15.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.40.15.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.40.15.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.40.15.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.40.15.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.40.15.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.40.15.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.40.15.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.40.15.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.40.15.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.40.15.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.40.15.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.40.15.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.40.15.tgz
	secret-controller	https://binary.mirantis.com/core/helm/secret-controller-1.40.15.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.40.15.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.40.15.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.40.15.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.40.15.tgz
	vsphere-vm-template-controller	https://binary.mirantis.com/core/helm/vsphere-vm-template-controller-1.40.15.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.40.15
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.40.15
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.40.15
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.40.15
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-6
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.40.15
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.40.15
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.40.15
	host-os-modules-controller ^Updated	mirantis.azurecr.io/core/host-os-modules-controller:1.40.15
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.40.15
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.40.15
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.40.15
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.40.15
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.40.15
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.40.15
	mcc-haproxy ^Updated	mirantis.azurecr.io/lcm/mcc-haproxy:v0.25.0-40-g890ffca
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.25.0-40-g890ffca
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.40.15
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.40.15
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.40.15
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.40.15
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.40.15
	registry ^Updated	mirantis.azurecr.io/lcm/registry:v2.8.1-10
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.40.15
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.40.15
	secret-controller ^Updated	mirantis.azurecr.io/core/secret-controller:1.40.15
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-10-g24a0d69
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.40.15
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.40.15
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.40.15
	vsphere-vm-template-controller ^Updated	mirantis.azurecr.io/core/vsphere-vm-template-controller:1.40.15

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts ^Updated	iam	https://binary.mirantis.com/core/helm/iam-1.40.15.tgz
Docker images	kubectl	mirantis.azurecr.io/stacklight/kubectl:1.22-20240501023013
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-ba8ada4-20240405150338
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.17-focal-20240523075821
	mcc-keycloak ^Updated	mirantis.azurecr.io/iam/mcc-keycloak:24.0.5-20240621131831

2.27.0¶

The Mirantis Container Cloud major release 2.27.0:

Introduces support for the Cluster release 17.2.0 that is based on the Cluster release 16.2.0 and represents Mirantis OpenStack for Kubernetes (MOSK) 24.2.
Introduces support for the Cluster release 16.2.0 that is based on Mirantis Container Runtime (MCR) 23.0.11 and Mirantis Kubernetes Engine (MKE) 3.7.8 with Kubernetes 1.27.
Does not support greenfield deployments on deprecated Cluster releases of the 17.1.x and 16.1.x series. Use the latest available Cluster releases of the series instead.

Caution

Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.27.0.

Enhancements¶

This section outlines new features and enhancements introduced in the Container Cloud release 2.27.0. For the list of enhancements delivered with the Cluster releases introduced by Container Cloud 2.27.0, see 17.2.0 and 16.2.0.

General availability for Ubuntu 22.04 on bare metal clusters¶

Implemented full support for Ubuntu 22.04 LTS (Jellyfish) as the default host operating system that now installs on non-MOSK bare metal management and managed clusters.

For MOSK:

Existing management clusters are automatically updated to Ubuntu 22.04 during cluster upgrade to Container Cloud 2.27.0 (Cluster release 16.2.0).
Greenfield deployments of management clusters are based on Ubuntu 22.04.
Existing and greenfield deployments of managed clusters are still based on Ubuntu 20.04. The support for Ubuntu 22.04 on this cluster type will be announced in one of the following releases.

Caution

Upgrading from Ubuntu 20.04 to 22.04 on existing deployments of Container Cloud managed clusters is not supported.

Learn more

Improvements in the day-2 management API for bare metal clusters¶

TechPreview

Enhanced the day-2 management API the bare metal provider with several key improvements:

Implemented the sysctl, package, and irqbalance configuration modules, which become available for usage after your management cluster upgrade to the Cluster release 16.2.0. These Container Cloud modules use the designated HostOSConfiguration object named mcc-modules to distingish them from custom modules.

Configuration modules allow managing the operating system of a bare metal host granularly without rebuilding the node from scratch. Such approach prevents workload evacuation and significantly reduces configuration time.
Optimized performance for faster, more efficient operations.
Enhanced user experience for easier and more intuitive interactions.
Resolved various internal issues to ensure smoother functionality.
Added comprehensive documentation, including concepts, guidelines, and recommendations for effective use of day-2 operations.

Learn more

Optimization of strict filtering for devices on bare metal clusters¶

Optimized the BareMetalHostProfile custom resource, which uses the strict byID filtering to target system disks using the byPath, serialNumber, and wwn reliable device options instead of the unpredictable byName naming format.

The optimization includes changes in admission-controller that now blocks the use of bmhp:spec:devices:by_name in new BareMetalHostProfile objects.

Learn more

Deprecation of SubnetPool and MetalLBConfigTemplate objects¶

As part of refactoring of the bare metal provider, deprecated the SubnetPool and MetalLBConfigTemplate objects. The objects will be completely removed from the product in one of the following releases.

Both objects are automatically migrated to the MetallbConfig object during cluster update to the Cluster release 17.2.0 or 16.2.0.

Learn more

Deprecation notes

The ClusterUpdatePlan object for a granular cluster update¶

TechPreview

Implemented the ClusterUpdatePlan custom resource to enable a granular step-by-step update of a managed cluster. The operator can control the update process by manually launching update stages using the commence flag. Between the update stages, a cluster remains functional from the perspective of cloud users and workloads.

A ClusterUpdatePlan object is automatically created by the respective Container Cloud provider when a new Cluster release becomes available for your cluster. This object contains a list of predefined self-descriptive update steps that are cluster-specific. These steps are defined in the spec section of the object with information about their impact on the cluster.

Learn more

Operations Guide: Granularly update a managed cluster

Update groups for worker machines¶

Implemented the UpdateGroup custom resource for creation of update groups for worker machines on managed clusters. The use of update groups provides enhanced control over update of worker machines. This feature decouples the concurrency settings from the global cluster level, providing update flexibility based on the workload characteristics of different worker machine sets.

Learn more

LCM Agent heartbeats¶

Implemented the same heartbeat model for the LCM Agent as Kubernetes uses for Nodes. This model allows reflecting the actual status of the LCM Agent when it fails. For visual representation, added the corresponding LCM Agent status to the Container Cloud web UI for clusters and machines, which reflects health status of the LCM agent along with its status of update to the version from the current Cluster release.

Learn more

Handling secret leftovers using secret-controller¶

Implemented secret-controller that runs on a management cluster and cleans up secret leftovers of credentials that are not cleaned up automatically after creation of new secrets. This controller replaces rhellicense-controller, proxy-controller, and byo-credentials-controller as well as partially replaces the functionality of license-controller and other credential controllers.

Note

You can change memory limits for secret-controller on a management cluster using the resources:limits parameter in the spec:providerSpec:value:kaas:management:helmReleases: section of the Cluster object.

Learn more

MOSK documentation: Underlay Kubernetes operations - Increase memory limits for cluster components

MariaDB backup for bare metal and vSphere providers¶

Implemented the capability to back up and restore MariaDB databases on management clusters for bare metal and vSphere providers. Also, added documentation on how to change the storage node for backups on clusters of these provider types.

Learn more

Operations Guide: Back up MariaDB on a management cluster

Addressed issues¶

The following issues have been addressed in the Mirantis Container Cloud release 2.27.0 along with the Cluster releases 17.2.0 and 16.2.0.

Note

This section provides descriptions of issues addressed since the last Container Cloud patch release 2.26.5.

For details on addressed issues in earlier patch releases since 2.26.0, which are also included into the major release 2.27.0, refer to 2.26.x patch releases.

[42304] [StackLight] Fixed the issue with failure of shard relocation in the OpenSearch cluster on large Container Cloud managed clusters.
[41890] [StackLight] Fixed the issue with Patroni failing to start because of the short default timeout.
[40020] [StackLight] Fixed the issue with rollover_policy not being applied to the current indices while updating the policy for the current system* and audit* data streams.
[41819] [Ceph] Fixed the issue with the graceful cluster reboot being blocked by active Ceph ClusterWorkloadLock objects.
[28865] [LCM] Fixed the issue with validation of the NTP configuration before cluster deployment. Now, deployment does not start until the NTP configuration is validated.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.27.0 including the Cluster releases 17.2.0 and 16.2.0.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[47202] Inspection error on bare metal hosts after dnsmasq restart¶

Note

Moving forward, the workaround for this issue will be moved from Release Notes to MOSK Troubleshooting Guide: Inspection error on bare metal hosts after dnsmasq restart.

The issue can occur when:

The dnsmasq pod was moved to another node.
DHCP subnets were changed, including addition or removal. In this case, the dhcpd container of the dnsmasq pod is restarted.

Caution

If changing or adding of DHCP subnets is required to bootstrap new nodes, wait after changing or adding DHCP subnets until the dnsmasq pod becomes ready, then create BareMetalHost objects.

To verify whether the nodes are affected:

Verify whether the BareMetalHost objects contain the inspection error:

kubectl get bmh -n <managed-cluster-namespace-name>

Example of system response:

NAME            STATE         CONSUMER        ONLINE   ERROR              AGE
test-master-1   provisioned   test-master-1   true                        9d
test-master-2   provisioned   test-master-2   true                        9d
test-master-3   provisioned   test-master-3   true                        9d
test-worker-1   provisioned   test-worker-1   true                        9d
test-worker-2   provisioned   test-worker-2   true                        9d
test-worker-3   inspecting                    true     inspection error   19h

Verify whether the dnsmasq pod was in Ready state when the inspection of the affected baremetal hosts (test-worker-3 in the example above) was started:

kubectl -n kaas get pod <dnsmasq-pod-name> -oyaml

Example of system response:

...
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-10-10T15:37:34Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2024-10-11T07:38:54Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2024-10-11T07:38:54Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2024-10-10T15:37:34Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://6dbcf2fc4b36ce4c549c9191ab01f72d0236c51d42947675302675e4bfaf4cdf
    image: docker-dev-kaas-virtual.artifactory-eu.mcp.mirantis.net/bm/baremetal-dnsmasq:base-2-28-alpine-20240812132650
    imageID: docker-dev-kaas-virtual.artifactory-eu.mcp.mirantis.net/bm/baremetal-dnsmasq@sha256:3dad3e278add18e69b2608e462691c4823942641a0f0e25e6811e703e3c23b3b
    lastState:
      terminated:
        containerID: containerd://816fcf079cd544acd74e312065de5b5ed4dbf1dc6159fefffff4f644b5e45987
        exitCode: 0
        finishedAt: "2024-10-11T07:38:35Z"
        reason: Completed
        startedAt: "2024-10-10T15:37:45Z"
    name: dhcpd
    ready: true
    restartCount: 2
    started: true
    state:
      running:
        startedAt: "2024-10-11T07:38:37Z"
  ...

In the system response above, the dhcpd container was not ready between "2024-10-11T07:38:35Z" and "2024-10-11T07:38:54Z".

Verify the affected baremetal host. For example:

kubectl get bmh -n managed-ns test-worker-3 -oyaml

Example of system response:

...
status:
  errorCount: 15
  errorMessage: Introspection timeout
  errorType: inspection error
  ...
  operationHistory:
    deprovision:
      end: null
      start: null
    inspect:
      end: null
      start: "2024-10-11T07:38:19Z"
    provision:
      end: null
      start: null
    register:
      end: "2024-10-11T07:38:19Z"
      start: "2024-10-11T07:37:25Z"

Workaround

Reboot the node using the IPMI reset or cycle command.
If the node fails to boot, remove the failed BareMetalHost object and create it again:
1. Remove BareMetalHost object. For example:
```
kubectl delete bmh -n managed-ns test-worker-3
```
2. Verify that the BareMetalHost object is removed:
```
kubectl get bmh -n managed-ns test-worker-3
```
3. Create a BareMetalHost object from the template. For example:
```
kubectl create -f bmhc-test-worker-3.yaml
kubectl create -f bmh-test-worker-3.yaml
```

[46245] Lack of access permissions for HOC and HOCM objects¶

Fixed in 2.28.0 (17.3.0 and 16.3.0)

kubectl --kubeconfig ~/.kube/mgmt-config get hocm

Error from server (Forbidden): hostosconfigurationmodules.kaas.mirantis.com is forbidden:
User "2d74348b-5669-4c65-af31-6c05dbedac5f" cannot list resource "hostosconfigurationmodules"
in API group "kaas.mirantis.com" at the cluster scope: access denied

Workaround:

Modify the global-admin role by adding a new entry with the following contents to the rules list:

kubectl edit clusterroles kaas-global-admin

- apiGroups: [kaas.mirantis.com]
  resources: [hostosconfigurationmodules]
  verbs: ['*']

For each Container Cloud project, modify the kaas-operator role by adding a new entry with the following contents to the rules list:

kubectl -n <projectName> edit roles kaas-operator

- apiGroups: [kaas.mirantis.com]
  resources: [hostosconfigurations]
  verbs: ['*']

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[41305] DHCP responses are lost between dnsmasq and dhcp-relay pods¶

Fixed in 2.28.0 (17.3.0 and 16.3.0)

The issue was observed on environments having management cluster nodes configured with a single L2 segment used for all network traffic (PXE and LCM/management networks).

To verify whether the cluster is affected:

Verify whether the dnsmasq and dhcp-relay pods run on the same node in the management cluster:

kubectl -n kaas get pods -o wide| grep -e "dhcp\|dnsmasq"

Example of system response:

dhcp-relay-7d85f75f76-5vdw2   2/2   Running   2 (36h ago)   36h   10.10.0.122     kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>
dnsmasq-8f4b484b4-slhbd       5/5   Running   1 (36h ago)   36h   10.233.123.75   kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>

If this is the case, proceed to the workaround below.

Workaround:

Make sure that at least two management cluster nodes are schedulable:

kubectl get node

Example of a positive system response:

NAME                                             STATUS   ROLES    AGE   VERSION
kaas-node-bcedb87b-b3ce-46a4-a4ca-ea3068689e40   Ready    master   37h   v1.27.10-mirantis-1
kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   Ready    master   37h   v1.27.10-mirantis-1
kaas-node-ad5a6f51-b98f-43c3-91d5-55fed3d0ff21   Ready    master   37h   v1.27.10-mirantis-1

Delete the dhcp-relay pod:

kubectl -n kaas delete pod <dhcp-relay-xxxxx>

Verify that the dnsmasq and dhcp-relay pods are scheduled into different nodes:

kubectl -n kaas get pods -o wide| grep -e "dhcp\|dnsmasq"

Example of a positive system response:

dhcp-relay-7d85f75f76-rkv03   2/2   Running   0             49s   10.10.0.121     kaas-node-bcedb87b-b3ce-46a4-a4ca-ea3068689e40   <none>   <none>
dnsmasq-8f4b484b4-slhbd       5/5   Running   1 (37h ago)   37h   10.233.123.75   kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

LCM¶

[39437] Failure to replace a master node on a Container Cloud cluster¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

During the replacement of a master node on a cluster of any type, the process may get stuck with Kubelet's NodeReady condition is Unknown in the machine status on the remaining master nodes.

As a workaround, log in on the affected node and run the following command:

docker restart ucp-kubelet

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

[30294] Replacement of a master node is stuck on the calico-node Pod start¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a master node on a cluster of any type, the calico-node Pod fails to start on a new node that has the same IP address as the node being replaced.

Workaround:

From a CLI with an MKE client bundle, create a shell alias to start calicoctl using the mirantis/ucp-dsinfo image:

Since MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/key.pem \
-e ETCD_CA_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/ca.pem \
-e ETCD_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/cert.pem \
-v /var/run/calico:/var/run/calico \
-v /var/lib/docker/volumes/ucp-kv-certs/_data:/var/lib/docker/volumes/ucp-kv-certs/_data:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl \
"

Before MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
-e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
-e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
-v /var/run/calico:/var/run/calico \
-v ucp-node-certs:/ucp-node-certs:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl --allow-version-mismatch \
"

In the above command, replace the following values with the corresponding settings of the affected cluster:

<etcdEndpoint> is the etcd endpoint defined in the Calico configuration file. For example, ETCD_ENDPOINTS=127.0.0.1:12378
<mkeVersion> is the MKE version installed on your cluster. For example, mirantis/ucp-dsinfo:3.5.7.

Verify the node list on the cluster:
```
kubectl get node
```
Compare this list with the node list in Calico to identify the old node:
```
calicoctl get node -o wide
```

Remove the old node from Calico:

calicoctl delete node kaas-node-<nodeID>

[5782] Manager machine fails to be deployed during node replacement¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a manager machine, the following problems may occur:

The system adds the node to Docker swarm but not to Kubernetes
The node Deployment gets stuck with failed RethinkDB health checks

Workaround:

Delete the failed node.
Wait for the MKE cluster to become healthy. To monitor the cluster status:
1. Log in to the MKE web UI as described in Connect to the Mirantis Kubernetes Engine web UI.
2. Monitor the cluster status as described in MKE Operations Guide: Monitor an MKE cluster with the MKE web UI.
Deploy a new node.

[5568] The calico-kube-controllers Pod fails to clean up resources¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During the unsafe or forced deletion of a manager machine running the calico-kube-controllers Pod in the kube-system namespace, the following issues occur:

The calico-kube-controllers Pod fails to clean up resources associated with the deleted node
The calico-node Pod may fail to start up on a newly created node if the machine is provisioned with the same IP address as the deleted machine had

As a workaround, before deletion of the node running the calico-kube-controllers Pod, cordon and drain the node:

kubectl cordon <nodeName>
kubectl drain <nodeName>

Ceph¶

[50566] Ceph upgrade is very slow during patch or major cluster update¶

Fixed in 2.29.3 (17.3.8, 16.3.8, and 16.4.3)

 Warning  Unhealthy  57s (x16 over 3m27s)  kubelet  Startup probe failed:
 ceph daemon health check failed with the following output:
> no valid command found; 10 closest matches:
> 0
> 1
> 2
> abort
> assert
> bluefs debug_inject_read_zeros
> bluefs files list
> bluefs stats
> bluestore bluefs device info [<alloc_size:int>]
> config diff
> admin_socket: invalid command

Workaround:

Complete the following steps during every patch or major cluster update of the Cluster releases 17.2.x, 17.3.x, and 17.4.x (until Ceph 18.2.5 becomes supported):

Plan extra time in the maintenance window for the patch cluster update.

Slow starts will still impact the update procedure, but after completing the following step, the recovery process noticeably shortens without affecting the overall cluster state and data responsiveness.
Select one of the following options:
- Before the cluster update, set the noout flag:
```
ceph osd set noout
```
  Once the Ceph OSDs image upgrade is done, unset the flag:
```
ceph osd unset noout
```
- Monitor the Ceph OSDs image upgrade. If the symptoms of slow start appear, set the noout flag as soon as possible. Once the Ceph OSDs image upgrade is done, unset the flag.

[42908] The ceph-exporter pods are present in the Ceph crash list¶

After a managed cluster update, the ceph-exporter pods are present in the ceph crash ls list while rook-ceph-exporter attempts to obtain the port that is still in use. The issue does not block the managed cluster update. Once the port becomes available, rook-ceph-exporter obtains the port and the issue disappears.

As a workaround, run ceph crash archive-all to remove ceph-exporter pods from the Ceph crash list.

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

StackLight¶

[44193] OpenSearch reaches 85% disk usage watermark affecting the cluster state¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

Caution

The issue and the provided workaround apply only for clusters on which OpenSearch and Prometheus utilize the same storage pool.

To verify that the cluster is affected:

Verify the result of the following formula:
```
0.8 × OpenSearch_PVC_Size_GB + Prometheus_PVC_Size_GB > 0.85 × Total_Storage_Capacity_GB
```
In the formula, define the following values:
OpenSearch_PVC_Size_GB
Derived from .values.elasticsearch.persistentVolumeUsableStorageSizeGB, defaulting to .values.elasticsearch.persistentVolumeClaimSize if unspecified. To obtain the OpenSearch PVC size:
kubectl -n <namespaceName> get cluster <clusterName> -o yaml |\ yq '.spec.providerSpec.value.helmReleases[] | select(.name == "stacklight") | .values.elasticsearch.persistentVolumeClaimSize '
Example of system response:
10000Gi
Prometheus_PVC_Size_GB
Sourced from .values.prometheusServer.persistentVolumeClaimSize. To obtain the Prometheus PVC size:
kubectl -n <namespaceName> get cluster <clusterName> -o yaml |\ yq '.spec.providerSpec.value.helmReleases[] | select(.name == "stacklight") | .values.prometheusServer.persistentVolumeClaimSize '
Example of system response:
4000Gi
Total_Storage_Capacity_GB
Total capacity of the OpenSearch PVCs. For LVP, the capacity of the storage pool. To obtain the total capacity:
kubectl get pvc -n stacklight -l app=opensearch-master \ -o custom-columns=NAME:.metadata.name,CAPACITY:.status.capacity.storage
The system response contains multiple outputs, one per opensearch-master node. Select the capacity for the affected node.
Note

Convert the values to GB if they are set in different units.

If the formula result is positive, it is an early indication that the cluster is affected.
Verify whether the OpenSearchClusterStatusWarning or OpenSearchClusterStatusCritical alert is firing. And if so, verify the following:
1. Log in to the OpenSearch web UI.
2. In Management -> Dev Tools, run the following command:
```
GET _cluster/allocation/explain
```
  The following system response indicates that the corresponding node is affected:
```
"explanation": "the node is above the low watermark cluster setting \
[cluster.routing.allocation.disk.watermark.low=85%], using more disk space \
than the maximum allowed [85.0%], actual free: [xx.xxx%]"
```
  Note
  
  The system response may contain even higher watermark percent than 85.0%, depending on the case.

Workaround:

Warning

The workaround implies adjustement of the retention threshold for OpenSearch. And depending on the new threshold, some old logs will be deleted.

Adjust or set .values.elasticsearch.persistentVolumeUsableStorageSizeGB to a lower value for the affection check formula to be non-positive. For configuration details, see MOSK Operations Guide: StackLight configuration parameters - OpenSearch.

Mirantis also recommends reserving some space for other PVCs using storage from the pool. Use the following formula to calculate the required space:
```
persistentVolumeUsableStorageSizeGB =
0.84 × ((1 - Reserved_Percentage - Filesystem_Reserve) ×
Total_Storage_Capacity_GB - Prometheus_PVC_Size_GB) /
0.8
```
In the formula, define the following values:
Reserved_Percentage
A user-defined variable that specifies what percentage of the total storage capacity should not be used by OpenSearch or Prometheus. This is used to reserve space for other components. It should be expressed as a decimal. For example, for 5% of reservation, Reserved_Percentage is 0.05. Mirantis recommends using 0.05 as a starting point.

Filesystem_Reserve
Percentage to deduct for filesystems that may reserve some portion of the available storage, which is marked as occupied. For example, for EXT4, it is 5% by default, so the value must be 0.05.

Prometheus_PVC_Size_GB
Sourced from .values.prometheusServer.persistentVolumeClaimSize.

Total_Storage_Capacity_GB
Total capacity of the OpenSearch PVCs. For LVP, the capacity of the storage pool. To obtain the total capacity:
kubectl get pvc -n stacklight -l app=opensearch-master \ -o custom-columns=NAME:.metadata.name,CAPACITY:.status.capacity.storage
The system response contains multiple outputs, one per opensearch-master node. Select the capacity for the affected node.
Note

Convert the values to GB if they are set in different units.

Calculation of above formula provides a maximum safe storage to allocate for .values.elasticsearch.persistentVolumeUsableStorageSizeGB. Use this formula as a reference for setting .values.elasticsearch.persistentVolumeUsableStorageSizeGB on a cluster.
Wait up to 15-20 mins for OpenSearch to perform the cleaning.
Verify that the cluster is not affected anymore using the procedure above.

[43164] Rollover policy is not added to indicies created without a policy¶

Fixed in 2.28.0 (17.3.0 and 16.3.0)

The initial index for the system* and audit* data streams can be created without any policy attached due to race condition.

One of indicators that the cluster is most likely affected is the KubeJobFailed alert firing for the elasticsearch-curator job and one or both of the following errors being present in elasticsearch-curator pods that remain in the Error status:

2024-05-31 13:16:04,459 ERROR   Failed to complete action: delete_indices.  \
<class 'curator.exceptions.FailedExecution'>: Exception encountered.  \
Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. \
Exception: RequestError(400, 'illegal_argument_exception', 'index [.ds-system-000001] \
is the write index for data stream [system] and cannot be deleted')

2024-05-31 13:16:04,459 ERROR   Failed to complete action: delete_indices.  \
<class 'curator.exceptions.FailedExecution'>: Exception encountered.  \
Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. \
Exception: RequestError(400, 'illegal_argument_exception', 'index [.ds-audit-000001] \
is the write index for data stream [audit] and cannot be deleted')

If the above mentioned alert and errors are present, an immediate action is required, because it indicates that the corresponding index size has already exceeded the space allocated for the index.

To verify that the cluster is affected:

Caution

Verify and apply the workaround to both index patterns, system and audit, separately.

If one of indices is affected, the second one is most likely affected as well. Although in rare cases, only one index may be affected.

kubectl exec -it pod/opensearch-master-0 -n stacklight -c opensearch -- bash

Verify whether the rollover policy is attached to the index with the 000001 number:

system:

curl localhost:9200/_plugins/_ism/explain/.ds-system-000001

audit:

curl localhost:9200/_plugins/_ism/explain/.ds-audit-000001

If the rollover policy is not attached, the cluster is affected. Examples of system responses in an affected cluster:

 {
  ".ds-system-000001": {
    "index.plugins.index_state_management.policy_id": null,
    "index.opendistro.index_state_management.policy_id": null,
    "enabled": null
  },
  "total_managed_indices": 0
}

{
  ".ds-audit-000001": {
    "index.plugins.index_state_management.policy_id": null,
    "index.opendistro.index_state_management.policy_id": null,
    "enabled": null
  },
  "total_managed_indices": 0
}

Workaround:

kubectl exec -it pod/opensearch-master-0 -n stacklight -c opensearch -- bash

Add the policy:

system:

curl -XPOST -H "Content-type: application/json" localhost:9200/_plugins/_ism/add/system* -d'{"policy_id":"system_rollover_policy"}'

audit:

curl -XPOST -H "Content-type: application/json" localhost:9200/_plugins/_ism/add/audit* -d'{"policy_id":"audit_rollover_policy"}'

Perform again the last step of the cluster verification procedure provided above and make sure that the policy is attached to the index.

Container Cloud web UI¶

[50181] Failure to deploy a compact cluster¶

To work around the issue, manually add the required labels using CLI. Once done, the cluster deployment resumes.

[50168] Inability to use a new project right after creation¶

A newly created project does not display all available tabs in the Container Cloud web UI and contains different access denied errors during first five minutes after creation.

To work around the issue, refresh the browser in five minutes after the project creation.

Components versions¶

The following table lists the major components and their versions delivered in Container Cloud 2.27.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Component	Application/Service	Version
Bare metal	baremetal-dnsmasq ^Updated	base-2-27-alpine-20240523143049
	baremetal-operator ^Updated	base-2-27-alpine-20240523142757
	baremetal-provider ^Updated	1.40.11
	bm-collective ^Updated	base-2-27-alpine-20240523143803
	cluster-api-provider-baremetal ^Updated	1.40.11
	ironic ^Updated	antelope-jammy-20240522120643
	ironic-inspector ^Updated	antelope-jammy-20240522120643
	ironic-prometheus-exporter	0.1-20240117102150
	kaas-ipam ^Updated	base-2-27-alpine-20240531082457
	kubernetes-entrypoint	v1.0.1-ba8ada4-20240405150338
	mariadb	10.6.17-focal-20240523075821
	metallb-controller ^Updated	v0.14.5-e86184d9-amd64
	metallb-speaker ^Updated	v0.14.5-e86184d9-amd64
	syslog-ng	base-alpine-20240129163811
Container Cloud	admission-controller ^Updated	1.40.11
	agent-controller ^Updated	1.40.11
	byo-cluster-api-controller ^Updated	1.40.11
	byo-credentials-controller ^Removed	n/a
	ceph-kcc-controller ^Updated	1.40.11
	cert-manager-controller	1.11.0-6
	cinder-csi-plugin	1.27.2-16
	client-certificate-controller ^Updated	1.40.11
	configuration-collector ^Updated	1.40.11
	csi-attacher	4.2.0-5
	csi-node-driver-registrar	2.7.0-5
	csi-provisioner	3.4.1-5
	csi-resizer	1.7.0-5
	csi-snapshotter	6.2.1-mcc-4
	event-controller ^Updated	1.40.11
	frontend ^Updated	1.40.12
	golang	1.21.7-alpine3.18
	iam-controller ^Updated	1.40.11
	kaas-exporter ^Updated	1.40.11
	kproxy ^Updated	1.40.11
	lcm-controller ^Updated	1.40.11
	license-controller ^Updated	1.40.11
	livenessprobe ^Updated	2.9.0-5
	machinepool-controller ^Updated	1.40.11
	mcc-haproxy ^Updated	0.25.0-37-gc15c97d
	metrics-server	0.6.3-7
	nginx ^Updated	1.40.11
	policy-controller ^New	1.40.11
	portforward-controller ^Updated	1.40.11
	proxy-controller ^Updated	1.40.11
	rbac-controller ^Updated	1.40.11
	registry	2.8.1-9
	release-controller ^Updated	1.40.11
	rhellicense-controller ^Removed	n/a
	scope-controller ^Updated	1.40.11
	secret-controller ^New	1.40.11
	storage-discovery ^Updated	1.40.11
	user-controller ^Updated	1.40.11
IAM	iam ^Updated	1.40.11
	mariadb	10.6.17-focal-20240523075821
	mcc-keycloak ^Updated	24.0.3-20240527150505
OpenStack ^Updated	host-os-modules-controller ^Updated	1.40.11
	openstack-cloud-controller-manager	v1.27.2-16
	openstack-cluster-api-controller	1.40.11
	openstack-provider	1.40.11
	os-credentials-controller	1.40.11
VMware vSphere	mcc-keepalived ^Updated	0.25.0-37-gc15c97d
	squid-proxy	0.0.1-10-g24a0d69
	vsphere-cloud-controller-manager	v1.27.0-6
	vsphere-cluster-api-controller ^Updated	1.40.11
	vsphere-credentials-controller ^Updated	1.40.11
	vsphere-csi-driver	v3.0.2-1
	vsphere-csi-syncer	v3.0.2-1
	vsphere-provider ^Updated	1.40.11
	vsphere-vm-template-controller ^Updated	1.40.11

Artifacts¶

This section lists the artifacts of components included in the Container Cloud release 2.27.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20240517093708
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20240517093708
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.40.11.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.40.11.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.40.11.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.40.11.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.40.11.tgz
	local-volume-provisioner	https://binary.mirantis.com/core/helm/local-volume-provisioner-1.40.11.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.40.11.tgz
Docker images	ambasador ^Updated	mirantis.azurecr.io/core/external/nginx:1.40.11
	baremetal-dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-27-alpine-20240523143049
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-2-27-alpine-20240523142757
	bm-collective ^Updated	mirantis.azurecr.io/bm/bm-collective:base-2-27-alpine-20240523143803
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.40.11
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:antelope-jammy-20240522120643
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:antelope-jammy-20240522120643
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20240117102150
	kaas-ipam ^Updated	mirantis.azurecr.io/bm/kaas-ipam:base-2-27-alpine-20240531082457
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-ba8ada4-20240405150338
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.17-focal-20240523075821
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.25.0-37-gc15c97d
	metallb-controller ^Updated	mirantis.azurecr.io/bm/metallb/controller:v0.14.5-e86184d9-amd64
	metallb-speaker ^Updated	mirantis.azurecr.io/bm/metallb/speaker:v0.14.5-e86184d9-amd64
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20240129163811

Core artifacts¶

Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.40.11.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.40.11.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.40.11.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.40.11.tgz
	byo-credentials-controller ^Removed	n/a
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.40.11.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.40.11.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.40.11.tgz
	cinder-csi-plugin	https://binary.mirantis.com/core/helm/cinder-csi-plugin-1.40.11.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.40.11.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.40.11.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.40.11.tgz
	host-os-modules-controller	https://binary.mirantis.com/core/helm/host-os-modules-controller-1.40.11.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.40.11.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.40.11.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.40.11.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.40.12.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.40.11.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.40.11.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.40.11.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.40.11.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.40.11.tgz
	metrics-server	https://binary.mirantis.com/core/helm/metrics-server-1.40.11.tgz
	openstack-cloud-controller-manager	https://binary.mirantis.com/core/helm/openstack-cloud-controller-manager-1.40.11.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.40.11.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.40.11.tgz
	policy-controller	https://binary.mirantis.com/core/helm/policy-controller-1.40.11.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.40.11.tgz
	proxy-controller ^Removed	n/a
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.40.11.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.40.11.tgz
	rhellicense-controller ^Removed	n/a
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.40.11.tgz
	secret-controller ^New	https://binary.mirantis.com/core/helm/secret-controller-1.40.11.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.40.11.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.40.11.tgz
	vsphere-cloud-controller-manager	https://binary.mirantis.com/core/helm/vsphere-cloud-controller-manager-1.40.11.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.40.11.tgz
	vsphere-csi-plugin	https://binary.mirantis.com/core/helm/vsphere-csi-plugin-1.40.11.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.40.11.tgz
	vsphere-vm-template-controller	https://binary.mirantis.com/core/helm/vsphere-vm-template-controller-1.40.11.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.40.11
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.40.11
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.40.11
	byo-credentials-controller ^Removed	n/a
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.40.11
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-6
	cinder-csi-plugin	mirantis.azurecr.io/lcm/kubernetes/cinder-csi-plugin:v1.27.2-16
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.40.11
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.40.11
	csi-attacher	mirantis.azurecr.io/lcm/k8scsi/csi-attacher:v4.2.0-5
	csi-node-driver-registrar	mirantis.azurecr.io/lcm/k8scsi/csi-node-driver-registrar:v2.7.0-5
	csi-provisioner	mirantis.azurecr.io/lcm/k8scsi/csi-provisioner:v3.4.1-5
	csi-resizer	mirantis.azurecr.io/lcm/k8scsi/csi-resizer:v1.7.0-5
	csi-snapshotter	mirantis.azurecr.io/lcm/k8scsi/csi-snapshotter:v6.2.1-mcc-4
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.40.11
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.40.12
	host-os-modules-controller ^Updated	mirantis.azurecr.io/core/host-os-modules-controller:1.40.11
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.40.11
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.40.11
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.40.11
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.40.11
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.40.11
	livenessprobe	mirantis.azurecr.io/lcm/k8scsi/livenessprobe:v2.9.0-5
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.40.11
	mcc-haproxy ^Updated	mirantis.azurecr.io/lcm/mcc-haproxy:v0.25.0-37-gc15c97d
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.25.0-37-gc15c97d
	metrics-server	mirantis.azurecr.io/lcm/metrics-server-amd64:v0.6.3-7
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.40.11
	openstack-cloud-controller-manager	mirantis.azurecr.io/lcm/kubernetes/openstack-cloud-controller-manager:v1.27.2-16
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.40.11
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.40.11
	policy-controller ^Updated	mirantis.azurecr.io/core/policy-controller:1.40.11
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.40.11
	proxy-controller ^Removed	n/a
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.40.11
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-9
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.40.11
	rhellicense-controller ^Removed	n/a
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.40.11
	secret-controller ^New	mirantis.azurecr.io/core/secret-controller:1.40.11
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-10-g24a0d69
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.40.11
	vsphere-cloud-controller-manager	mirantis.azurecr.io/lcm/kubernetes/vsphere-cloud-controller-manager:v1.27.0-6
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.40.11
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.40.11
	vsphere-csi-driver	mirantis.azurecr.io/lcm/kubernetes/vsphere-csi-driver:v3.0.2-1
	vsphere-csi-syncer	mirantis.azurecr.io/lcm/kubernetes/vsphere-csi-syncer:v3.0.2-1
	vsphere-vm-template-controller ^Updated	mirantis.azurecr.io/core/vsphere-vm-template-controller:1.40.11

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts ^Updated	iam	https://binary.mirantis.com/core/helm/iam-1.40.11.tgz
Docker images	kubectl	mirantis.azurecr.io/stacklight/kubectl:1.22-20240501023013
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-ba8ada4-20240405150338
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.17-focal-20240523075821
	mcc-keycloak ^Updated	mirantis.azurecr.io/iam/mcc-keycloak:24.0.3-20240527150505

Security notes¶

In total, since Container Cloud 2.26.0, in 2.27.0, 408 Common Vulnerabilities and Exposures (CVE) have been fixed: 26 of critical and 382 of high severity.

The table below includes the total numbers of addressed unique and common vulnerabilities and exposures (CVE) by product component since the 2.26.5 patch release. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	Critical	High	Total
Kaas core	Unique	0	7	7
Kaas core	Common	0	13	13
StackLight	Unique	4	14	18
StackLight	Common	4	25	29

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 24.2: Security notes.

Update notes¶

This section describes the specific actions you as a cloud operator need to complete before or after your Container Cloud cluster update to the Cluster releases 17.2.0 or 16.2.0.

Consider this information as a supplement to the generic update procedures published in Operations Guide: Automatic upgrade of a management cluster and Update a managed cluster.

Updated scheme for patch Cluster releases¶

Starting from Container Cloud 2.26.5, Mirantis introduces a new update scheme allowing for the update path flexibility. For details, see Patch update schemes before and since 2.26.5. For details on MOSK update scheme, refer to MOSK documentation: Update notes.

For those clusters that update between only major versions, the update scheme remains unchaged.

Caution

In Container Cloud patch releases 2.27.1 and 2.27.2, only the 16.2.x patch Cluster releases will be delivered with an automatic update of management clusters and the possibility to update non-MOSK managed clusters.

In parallel, 2.27.1 and 2.27.2 will include new 16.1.x and 17.1.x patches for MOSK 24.1.x. And the first 17.2.x patch Cluster release for MOSK 24.2.x will be delivered in 2.27.3. For details, see MOSK documentation: Update path for 24.1 and 24.2 series.

Pre-update actions¶

Update bird configuration on BGP-enabled bare metal clusters¶

Note

If you have already completed the below procedure after updating your clusters to Container Cloud 2.26.0 (Cluster releases 17.1.0 or 16.1.0), skip this subsection.

Container Cloud 2.26.0 introduced the bird daemon update from v1.6.8 to v2.0.7 on master nodes if BGP is used for BGP announcement of the cluster API load balancer address.

Configuration files for bird v1.x are not fully compatible with those for bird v2.x. Therefore, if you used BGP announcement of cluster API LB address on a deployment based on Cluster releases 17.0.0 or 16.0.0, update bird configuration files to fit bird v2.x using configuration examples provided in the API Reference: MultirRackCluster section.

Review and adjust the storage parameters for OpenSearch¶

Note

If you have already completed the below procedure after updating your clusters to Container Cloud 2.26.0 (Cluster releases 17.1.0 or 16.1.0), skip this subsection.

To prevent underused or overused storage space, review your storage space parameters for OpenSearch on the StackLight cluster:

Review the value of elasticsearch.persistentVolumeClaimSize and the real storage available on volumes.
Decide whether you have to additionally set elasticsearch.persistentVolumeUsableStorageSizeGB.

For both parameters description, see MOSK Operations Guide: StackLight configuration parameters - OpenSearch.

Post-update actions¶

Prepare for changing label values in Ceph metrics used in customizations¶

Note

If you do not use Ceph metrics in any customizations, for example, custom alerts, Grafana dashboards, or queries in custom workloads, skip this section.

After deprecating the performance metric exporter that is integrated into the Ceph Manager daemon for the sake of the dedicated Ceph Exporter daemon in Container Cloud 2.27.0, you may need to prepare for updating values of several labels in Ceph metrics if you use them in any customizations such as custom alerts, Grafana dashboards, or queries in custom tools. These labels will be changed in Container Cloud 2.28.0 (Cluster releases 16.3.0 and 17.3.0).

Note

Names of metrics will not be changed, no metrics will be removed.

Values of the job labels will be changed from rook-ceph-mgr to prometheus-rook-exporter for all Ceph metrics moved to Ceph Exporter. The full list of moved metrics is presented below.
Values of the instance labels will be changed from the metric endpoint of Ceph Manager with port 9283 to the metric endpoint of Ceph Exporter with port 9926 for all Ceph metrics moved to Ceph Exporter. The full list of moved metrics is presented below.
Values of the instance_id labels of Ceph metrics from the RADOS Gateway (RGW) daemons will be changed from the daemon GID to the daemon subname. For example, instead of instance_id="<RGW_PROCESS_GID>", the instance_id="a" (ceph_rgw_qlen{instance_id="a"}) will be used. The list of moved Ceph RGW metrics is presented below.

See also

2.26.5¶

The Container Cloud patch release 2.26.5, which is based on the 2.26.0 major release, provides the following updates:

Support for the patch Cluster releases 16.1.5 and 17.1.5 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 24.1.5.
Bare metal: update of Ubuntu mirror from 20.04~20240502102020 to 20.04~20240517090228 along with update of minor kernel version from 5.15.0-105-generic to 5.15.0-107-generic.
Security fixes for CVEs in images.
Bug fixes.

This patch release also supports the latest major Cluster releases 17.1.0 and 16.1.0. And it does not support greenfield deployments based on deprecated Cluster releases. Use the latest available Cluster release instead.

For main deliverables of the parent Container Cloud release of 2.26.5, refer to 2.26.0.

Security notes¶

The table below includes the total numbers of addressed unique and common CVEs in images by product component since the Container Cloud 2.26.4 patch release. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	Critical	High	Total
Ceph	Unique	0	1	1
Ceph	Common	0	3	3
Kaas core	Unique	0	5	5
Kaas core	Common	0	12	12
StackLight	Unique	1	3	4
StackLight	Common	2	6	8

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 24.1.5: Security notes.

Addressed issues¶

The following issues have been addressed in the Container Cloud patch release 2.26.5 along with the patch Cluster releases 17.1.5 and 16.1.5.

[42408] [bare metal] Fixed the issue with old versions of system packages, including kernel, remaining on the manager nodes after cluster update.
[41540] [LCM] Fixed the issue with lcm-agent failing to grab storage information on a host and leaving lcmmachine.status.hostinfo.hardware empty due to issues with managing physical NVME devices.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.26.4 including the Cluster releases 17.1.5 and 16.1.5.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[46245] Lack of access permissions for HOC and HOCM objects¶

Fixed in 2.28.0 (17.3.0 and 16.3.0)

kubectl --kubeconfig ~/.kube/mgmt-config get hocm

Error from server (Forbidden): hostosconfigurationmodules.kaas.mirantis.com is forbidden:
User "2d74348b-5669-4c65-af31-6c05dbedac5f" cannot list resource "hostosconfigurationmodules"
in API group "kaas.mirantis.com" at the cluster scope: access denied

Workaround:

Modify the global-admin role by adding a new entry with the following contents to the rules list:

kubectl edit clusterroles kaas-global-admin

- apiGroups: [kaas.mirantis.com]
  resources: [hostosconfigurationmodules]
  verbs: ['*']

For each Container Cloud project, modify the kaas-operator role by adding a new entry with the following contents to the rules list:

kubectl -n <projectName> edit roles kaas-operator

- apiGroups: [kaas.mirantis.com]
  resources: [hostosconfigurations]
  verbs: ['*']

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[41305] DHCP responses are lost between dnsmasq and dhcp-relay pods¶

Fixed in 2.28.0 (17.3.0 and 16.3.0)

The issue was observed on environments having management cluster nodes configured with a single L2 segment used for all network traffic (PXE and LCM/management networks).

To verify whether the cluster is affected:

Verify whether the dnsmasq and dhcp-relay pods run on the same node in the management cluster:

kubectl -n kaas get pods -o wide| grep -e "dhcp\|dnsmasq"

Example of system response:

dhcp-relay-7d85f75f76-5vdw2   2/2   Running   2 (36h ago)   36h   10.10.0.122     kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>
dnsmasq-8f4b484b4-slhbd       5/5   Running   1 (36h ago)   36h   10.233.123.75   kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>

If this is the case, proceed to the workaround below.

Workaround:

Make sure that at least two management cluster nodes are schedulable:

kubectl get node

Example of a positive system response:

NAME                                             STATUS   ROLES    AGE   VERSION
kaas-node-bcedb87b-b3ce-46a4-a4ca-ea3068689e40   Ready    master   37h   v1.27.10-mirantis-1
kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   Ready    master   37h   v1.27.10-mirantis-1
kaas-node-ad5a6f51-b98f-43c3-91d5-55fed3d0ff21   Ready    master   37h   v1.27.10-mirantis-1

Delete the dhcp-relay pod:

kubectl -n kaas delete pod <dhcp-relay-xxxxx>

Verify that the dnsmasq and dhcp-relay pods are scheduled into different nodes:

kubectl -n kaas get pods -o wide| grep -e "dhcp\|dnsmasq"

Example of a positive system response:

dhcp-relay-7d85f75f76-rkv03   2/2   Running   0             49s   10.10.0.121     kaas-node-bcedb87b-b3ce-46a4-a4ca-ea3068689e40   <none>   <none>
dnsmasq-8f4b484b4-slhbd       5/5   Running   1 (37h ago)   37h   10.233.123.75   kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

LCM¶

[39437] Failure to replace a master node on a Container Cloud cluster¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

During the replacement of a master node on a cluster of any type, the process may get stuck with Kubelet's NodeReady condition is Unknown in the machine status on the remaining master nodes.

As a workaround, log in on the affected node and run the following command:

docker restart ucp-kubelet

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

[30294] Replacement of a master node is stuck on the calico-node Pod start¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a master node on a cluster of any type, the calico-node Pod fails to start on a new node that has the same IP address as the node being replaced.

Workaround:

From a CLI with an MKE client bundle, create a shell alias to start calicoctl using the mirantis/ucp-dsinfo image:

Since MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/key.pem \
-e ETCD_CA_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/ca.pem \
-e ETCD_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/cert.pem \
-v /var/run/calico:/var/run/calico \
-v /var/lib/docker/volumes/ucp-kv-certs/_data:/var/lib/docker/volumes/ucp-kv-certs/_data:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl \
"

Before MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
-e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
-e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
-v /var/run/calico:/var/run/calico \
-v ucp-node-certs:/ucp-node-certs:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl --allow-version-mismatch \
"

In the above command, replace the following values with the corresponding settings of the affected cluster:

<etcdEndpoint> is the etcd endpoint defined in the Calico configuration file. For example, ETCD_ENDPOINTS=127.0.0.1:12378
<mkeVersion> is the MKE version installed on your cluster. For example, mirantis/ucp-dsinfo:3.5.7.

Verify the node list on the cluster:
```
kubectl get node
```
Compare this list with the node list in Calico to identify the old node:
```
calicoctl get node -o wide
```

Remove the old node from Calico:

calicoctl delete node kaas-node-<nodeID>

[5782] Manager machine fails to be deployed during node replacement¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a manager machine, the following problems may occur:

The system adds the node to Docker swarm but not to Kubernetes
The node Deployment gets stuck with failed RethinkDB health checks

Workaround:

Delete the failed node.
Wait for the MKE cluster to become healthy. To monitor the cluster status:
1. Log in to the MKE web UI as described in Connect to the Mirantis Kubernetes Engine web UI.
2. Monitor the cluster status as described in MKE Operations Guide: Monitor an MKE cluster with the MKE web UI.
Deploy a new node.

[5568] The calico-kube-controllers Pod fails to clean up resources¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During the unsafe or forced deletion of a manager machine running the calico-kube-controllers Pod in the kube-system namespace, the following issues occur:

The calico-kube-controllers Pod fails to clean up resources associated with the deleted node
The calico-node Pod may fail to start up on a newly created node if the machine is provisioned with the same IP address as the deleted machine had

As a workaround, before deletion of the node running the calico-kube-controllers Pod, cordon and drain the node:

kubectl cordon <nodeName>
kubectl drain <nodeName>

Ceph¶

[41819] Graceful cluster reboot is blocked by the Ceph ClusterWorkloadLocks¶

Fixed in 2.27.0 (17.2.0 and 16.2.0)

During graceful reboot of a cluster with Ceph enabled, the reboot is blocked with the following message in the MiraCephMaintenance object status:

message: ClusterMaintenanceRequest found, Ceph Cluster is not ready to upgrade,
 delaying cluster maintenance

As a workaround, add the following snippet to the cephFS section under metadataServer in the spec section of <kcc-name>.yaml in the Ceph cluster:

cephClusterSpec:
  sharedFilesystem:
    cephFS:
    - name: cephfs-store
      metadataServer:
        activeCount: 1
        healthCheck:
          livenessProbe:
            probe:
              failureThreshold: 5
              initialDelaySeconds: 30
              periodSeconds: 30
              successThreshold: 1
              timeoutSeconds: 5

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

StackLight¶

[42304] Failure of shard relocation in the OpenSearch cluster¶

Fixed in 17.2.0, 16.2.0, 17.1.6, 16.1.6

On large managed clusters, shard relocation may fail in the OpenSearch cluster with the yellow or red status of the OpenSearch cluster. The characteristic symptom of the issue is that in the stacklight namespace, the statefulset.apps/opensearch-master containers are experiencing throttling with the KubeContainersCPUThrottlingHigh alert firing for the following set of labels:

{created_by_kind="StatefulSet",created_by_name="opensearch-master",namespace="stacklight"}

Caution

The throttling that OpenSearch is experiencing may be a temporary situation, which may be related, for example, to a peaky load and the ongoing shards initialization as part of disaster recovery or after node restart. In this case, Mirantis recommends waiting until initialization of all shards is finished. After that, verify the cluster state and whether throttling still exists. And only if throttling does not disappear, apply the workaround below.

To verify that the initialization of shards is ongoing:

kubectl exec -it pod/opensearch-master-0 -n stacklight -c opensearch -- bash

curl "http://localhost:9200/_cat/shards" | grep INITIALIZING

Example of system response:

.ds-system-000072    2 r INITIALIZING    10.232.182.135 opensearch-master-1
.ds-system-000073    1 r INITIALIZING    10.232.7.145   opensearch-master-2
.ds-system-000073    2 r INITIALIZING    10.232.182.135 opensearch-master-1
.ds-audit-000001     2 r INITIALIZING    10.232.7.145   opensearch-master-2

The system response above indicates that shards from the .ds-system-000072, .ds-system-000073, and .ds-audit-000001 indicies are in the INITIALIZING state. In this case, Mirantis recommends waiting until this process is finished, and only then consider changing the limit.

You can additionally analyze the exact level of throttling and the current CPU usage on the Kubernetes Containers dashboard in Grafana.

Workaround:

Verify the currently configured CPU requests and limits for the opensearch containers:

kubectl -n stacklight get statefulset.apps/opensearch-master -o jsonpath="{.spec.template.spec.containers[?(@.name=='opensearch')].resources}"

Example of system response:

{"limits":{"cpu":"600m","memory":"8Gi"},"requests":{"cpu":"500m","memory":"6Gi"}}

In the example above, the CPU request is 500m and the CPU limit is 600m.

Increase the CPU limit to a reasonably high number.

For example, the default CPU limit for the clusters with the clusterSize:large parameter set was increased from 8000m to 12000m for StackLight in Container Cloud 2.27.0 (Cluster releases 17.2.0 and 16.2.0).

Note

For details, on the clusterSize parameter, see MOSK Operations Guide: StackLight configuration parameters - Cluster size.

If the defaults are already overridden on the affected cluster using the resourcesPerClusterSize or resources parameters as described in MOSK Operations Guide: StackLight configuration parameters - Resource limits, then the exact recommended number depends on the currently set limit.

Mirantis recommends increasing the limit by 50%. If it does not resolve the issue, another increase iteration will be required.
When you select the required CPU limit, increase it as described in MOSK Operations Guide: StackLight configuration parameters - Resource limits.

If the CPU limit for the opensearch component is already set, increase it in the Cluster object for the opensearch parameter. Otherwise, the default StackLight limit is used. In this case, increase the CPU limit for the opensearch component using the resources parameter.
Wait until all opensearch-master pods are recreated with the new CPU limits and become running and ready.

To verify the current CPU limit for every opensearch container in every opensearch-master pod separately:
```
kubectl -n stacklight get pod/opensearch-master-<podSuffixNumber> -o jsonpath="{.spec.containers[?(@.name=='opensearch')].resources}"
```
In the command above, replace <podSuffixNumber> with the name of the pod suffix. For example, pod/opensearch-master-0 or pod/opensearch-master-2.

Example of system response:
```
{"limits":{"cpu":"900m","memory":"8Gi"},"requests":{"cpu":"500m","memory":"6Gi"}}
```
The waiting time may take up to 20 minutes depending on the cluster size.

If the issue is fixed, the KubeContainersCPUThrottlingHigh alert stops firing immediately, while OpenSearchClusterStatusWarning or OpenSearchClusterStatusCritical can still be firing for some time during shard relocation.

If the KubeContainersCPUThrottlingHigh alert is still firing, proceed with another iteration of the CPU limit increase.

[40020] Rollover policy update is not appllied to the current index¶

Fixed in 17.2.0, 16.2.0, 17.1.6, 16.1.6

While updating rollover_policy for the current system* and audit* data streams, the update is not applied to indices.

2024-05-31 13:16:04,459 ERROR   Failed to complete action: delete_indices.  <class 'curator.exceptions.FailedExecution'>: Exception encountered.  Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: RequestError(400, 'illegal_argument_exception', 'index [.ds-audit-000001] is the write index for data stream [audit] and cannot be deleted')

2024-05-31 13:16:04,459 ERROR   Failed to complete action: delete_indices.  <class 'curator.exceptions.FailedExecution'>: Exception encountered.  Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: RequestError(400, 'illegal_argument_exception', 'index [.ds-system-000001] is the write index for data stream [system] and cannot be deleted')

Note

Instead of .ds-audit-000001 or .ds-system-000001 index names, similar names can be present with the same prefix but different suffix numbers.

If the above mentioned alert and errors are present, an immediate action is required, because it indicates that the corresponding index size has already exceeded the space allocated for the index.

To verify that the cluster is affected:

Caution

Verify and apply the workaround to both index patterns, system and audit, separately.

If one of indices is affected, the second one is most likely affected as well. Although in rare cases, only one index may be affected.

kubectl exec -it pod/opensearch-master-0 -n stacklight -c opensearch -- bash

Verify that the rollover policy is present:
- system:
```
curl localhost:9200/_plugins/_ism/policies/system_rollover_policy
```
- audit:
```
curl localhost:9200/_plugins/_ism/policies/audit_rollover_policy
```
The cluster is affected if the rollover policy is missing. Otherwise, proceed to the following step.

Verify the system response from the previous step. For example:

{"_id":"system_rollover_policy","_version":7229,"_seq_no":42362,"_primary_term":28,"policy":{"policy_id":"system_rollover_policy","description":"system index rollover policy.","last_updated_time":1708505222430,"schema_version":19,"error_notification":null,"default_state":"rollover","states":[{"name":"rollover","actions":[{"retry":{"count":3,"backoff":"exponential","delay":"1m"},"rollover":{"min_size":"14746mb","copy_alias":false}}],"transitions":[]}],"ism_template":[{"index_patterns":["system*"],"priority":200,"last_updated_time":1708505222430}]}}

Verify and capture the following items separately for every policy:

The _seq_no and _primary_term values
The rollover policy threshold, which is defined in policy.states[0].actions[0].rollover.min_size

List indices:

system:

curl localhost:9200/_cat/indices | grep system

Example of system response:

[...]
green open .ds-system-000001   FjglnZlcTKKfKNbosaE9Aw 2 1 1998295  0   1gb 507.9mb

audit:

curl localhost:9200/_cat/indices | grep audit

Example of system response:

[...]
green open .ds-audit-000001   FjglnZlcTKKfKNbosaE9Aw 2 1 1998295  0   1gb 507.9mb

Select the index with the highest number and verify the rollover policy attached to the index:
- system:
```
curl localhost:9200/_plugins/_ism/explain/.ds-system-000001
```
- audit:
```
curl localhost:9200/_plugins/_ism/explain/.ds-audit-000001
```
- If the rollover policy is not attached, the cluster is affected.
- If the rollover policy is attached but _seq_no and _primary_term numbers do not match the previously captured ones, the cluster is affected.
- If the index size drastically exceeds the defined threshold of the rollover policy (which is the previously captured min_size), the cluster is most probably affected.

Workaround:

kubectl exec -it pod/opensearch-master-0 -n stacklight -c opensearch -- bash

If the policy is attached to the index but has different _seq_no and _primary_term, remove the policy from the index:

Note

Use the index with the highest number in the name, which was captured during verification procedure.
- system:
```
curl -XPOST localhost:9200/_plugins/_ism/remove/.ds-system-000001
```
- audit:
```
curl -XPOST localhost:9200/_plugins/_ism/remove/.ds-audit-000001
```

Re-add the policy:

system:

curl -XPOST -H "Content-type: application/json" localhost:9200/_plugins/_ism/add/system* -d'{"policy_id":"system_rollover_policy"}'

audit:

curl -XPOST -H "Content-type: application/json" localhost:9200/_plugins/_ism/add/audit* -d'{"policy_id":"audit_rollover_policy"}'

Perform again the last step of the cluster verification procedure provided above and make sure that the policy is attached to the index and has the same _seq_no and _primary_term.

If the index size drastically exceeds the defined threshold of the rollover policy (which is the previously captured min_size), wait up to 15 minutes and verify that the additional index is created with the consecutive number in the index name. For example:
- system: if you applied changes to .ds-system-000001, wait until .ds-system-000002 is created.
- audit: if you applied changes to .ds-audit-000001, wait until .ds-audit-000002 is created.
If such index is not created, escalate the issue to Mirantis support.

Update notes¶

This section describes the specific actions you as a cloud operator need to complete before or after your Container Cloud cluster update to the Cluster releases 17.1.5 or 16.1.5.

Consider this information as a supplement to the generic update procedures published in Operations Guide: Automatic upgrade of a management cluster and Update a managed cluster.

Update scheme for patch Cluster releases¶

To improve user update experience and make the update path more flexible, Container Cloud is introducing a new scheme of updating between patch Cluster releases. More specifically, Container Cloud intends to ultimately provide a possibility to update to any newer patch version within single series at any point of time. The patch version downgrade is not supported.

Though, in some cases, Mirantis may request to update to some specific patch version in the series to be able to update to the next major series. This may be necessary due to the specifics of technical content already released or planned for the release. For possible update paths in MOSK in 24.1 and 24.2 series, see MOSK documentation: Cluster update scheme.

The exact number of patch releases for the 16.1.x and 17.1.x series is yet to be confirmed, but the current target is 7 releases.

Note

The management cluster update scheme remains the same. A management cluster obtains the new product version automatically after release.

See also

Patch releases
MOSK documentation:

Post-update actions¶

Delete ‘HostOSConfiguration’ objects on baremetal-based clusters¶

If you use the HostOSConfiguration and HostOSConfigurationModules custom resources for the bare metal provider, which are available in the Technology Preview scope in Container Cloud 2.26.x, delete all HostOSConfiguration objects right after update of your managed cluster to the Cluster release 17.1.5 or 16.1.5, before automatic upgrade of the management cluster to Container Cloud 2.27.0 (Cluster release 16.2.0). After the upgrade, you can recreate the required objects using the updated parameters.

This precautionary step prevents re-processing and re-applying of existing configuration, which is defined in HostOSConfiguration objects, during management cluster upgrade to 2.27.0. Such behavior is caused by changes in the HostOSConfiguration API introduced in 2.27.0.

Configure Kubernetes auditing and profiling for log rotation¶

Note

Skip this procedure if you have already completed it after updating your managed cluster to Container Cloud 2.26.4 (Cluster release 17.1.4 or 16.1.4).

After the MKE update to 3.7.8, if you are going to enable or already enabled Kubernetes auditing and profiling on your managed or management cluster, keep in mind that enabling audit log rotation requires an additional step. Set the following options in the MKE configuration file after enabling auditing and profiling:

[cluster_config]
  kube_api_server_audit_log_maxage=30
  kube_api_server_audit_log_maxbackup=10
  kube_api_server_audit_log_maxsize=10

For the configuration procedure, see MKE documentation: Configure an existing MKE cluster.

While using this procedure, replace the command to upload the newly edited MKE configuration file with the following one:

curl --silent --insecure -X PUT -H "X-UCP-Allow-Restricted-API: i-solemnly-swear-i-am-up-to-no-good" -H "accept: application/toml" -H "Authorization: Bearer $AUTHTOKEN" --upload-file 'mke-config.toml' https://$MKE_HOST/api/ucp/config-toml

The value for MKE_HOST has the <loadBalancerHost>:6443 format, where loadBalancerHost is the corresponding field in the cluster status.
The value for MKE_PASSWORD is taken from the ucp-admin-password-<clusterName> secret in the cluster namespace of the management cluster.
The value for MKE_USERNAME is always admin.

Artifacts¶

This section lists the artifacts of components included in the Container Cloud patch release 2.26.5. For artifacts of the Cluster releases introduced in 2.26.5, see patch Cluster releases 17.1.5 and 16.1.5.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries ^Updated	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20240517093708
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20240517093708
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.39.28.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.39.28.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.39.28.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.39.28.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.39.28.tgz
	local-volume-provisioner	https://binary.mirantis.com/core/helm/local-volume-provisioner-1.39.28.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.39.28.tgz
Docker images	ambasador ^Updated	mirantis.azurecr.io/core/external/nginx:1.39.28
	baremetal-dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-26-alpine-20240523095922
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-2-26-alpine-20240523095601
	bm-collective	mirantis.azurecr.io/bm/bm-collective:base-2-26-alpine-20240408142218
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.39.28
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:yoga-jammy-20240522120640
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:yoga-jammy-20240522120640
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20240117102150
	kaas-ipam	mirantis.azurecr.io/bm/kaas-ipam:base-2-26-alpine-20240408150853
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-ba8ada4-20240405150338
	mariadb ^Updated	mirantis.azurecr.io/general/mariadb:10.6.17-focal-20240523075821
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.24.0-47-gf77368e
	metallb-controller	mirantis.azurecr.io/bm/metallb/controller:v0.13.12-ef4c9453-amd64
	metallb-speaker	mirantis.azurecr.io/bm/metallb/speaker:v0.13.12-ef4c9453-amd64
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20240129163811

Core artifacts¶

Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.39.28.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.39.28.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.39.28.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.39.28.tgz
	byo-credentials-controller	https://binary.mirantis.com/core/helm/byo-credentials-controller-1.39.28.tgz
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.39.28.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.39.28.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.39.28.tgz
	cinder-csi-plugin	https://binary.mirantis.com/core/helm/cinder-csi-plugin-1.39.28.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.39.28.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.39.28.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.39.28.tgz
	host-os-modules-controller	https://binary.mirantis.com/core/helm/host-os-modules-controller-1.39.28.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.39.28.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.39.28.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.39.28.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.39.28.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.39.28.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.39.28.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.39.28.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.39.28.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.39.28.tgz
	metrics-server	https://binary.mirantis.com/core/helm/metrics-server-1.39.28.tgz
	openstack-cloud-controller-manager	https://binary.mirantis.com/core/helm/openstack-cloud-controller-manager-1.39.28.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.39.28.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.39.28.tgz
	policy-controller	https://binary.mirantis.com/core/helm/policy-controller-1.39.28.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.39.28.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.39.28.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.39.28.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.39.28.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.39.28.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.39.28.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.39.28.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.39.28.tgz
	vsphere-cloud-controller-manager	https://binary.mirantis.com/core/helm/vsphere-cloud-controller-manager-1.39.28.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.39.28.tgz
	vsphere-csi-plugin	https://binary.mirantis.com/core/helm/vsphere-csi-plugin-1.39.28.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.39.28.tgz
	vsphere-vm-template-controller	https://binary.mirantis.com/core/helm/vsphere-vm-template-controller-1.39.28.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.39.28
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.39.28
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.39.28
	byo-credentials-controller ^Updated	mirantis.azurecr.io/core/byo-credentials-controller:1.39.28
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.39.28
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-6
	cinder-csi-plugin	mirantis.azurecr.io/lcm/kubernetes/cinder-csi-plugin:v1.27.2-16
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.39.28
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.39.28
	csi-attacher	mirantis.azurecr.io/lcm/k8scsi/csi-attacher:v4.2.0-5
	csi-node-driver-registrar	mirantis.azurecr.io/lcm/k8scsi/csi-node-driver-registrar:v2.7.0-5
	csi-provisioner	mirantis.azurecr.io/lcm/k8scsi/csi-provisioner:v3.4.1-5
	csi-resizer	mirantis.azurecr.io/lcm/k8scsi/csi-resizer:v1.7.0-5
	csi-snapshotter	mirantis.azurecr.io/lcm/k8scsi/csi-snapshotter:v6.2.1-mcc-4
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.39.28
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.39.28
	host-os-modules-controller ^Updated	mirantis.azurecr.io/core/host-os-modules-controller:1.39.28
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.39.28
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.39.28
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.39.28
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.39.28
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.39.28
	livenessprobe	mirantis.azurecr.io/lcm/k8scsi/livenessprobe:v2.9.0-5
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.39.28
	mcc-haproxy	mirantis.azurecr.io/lcm/mcc-haproxy:v0.24.0-47-gf77368e
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.24.0-47-gf77368e
	metrics-server	mirantis.azurecr.io/lcm/metrics-server-amd64:v0.6.3-7
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.39.28
	openstack-cloud-controller-manager	mirantis.azurecr.io/lcm/kubernetes/openstack-cloud-controller-manager:v1.27.2-16
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.39.28
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.39.28
	policy-controller ^Updated	mirantis.azurecr.io/core/policy-controller:1.39.28
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.39.28
	proxy-controller ^Updated	mirantis.azurecr.io/core/proxy-controller:1.39.28
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.39.28
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-9
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.39.28
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.39.28
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.39.28
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-10-g24a0d69
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.39.28
	vsphere-cloud-controller-manager	mirantis.azurecr.io/lcm/kubernetes/vsphere-cloud-controller-manager:v1.27.0-6
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.39.28
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.39.28
	vsphere-csi-driver	mirantis.azurecr.io/lcm/kubernetes/vsphere-csi-driver:v3.0.2-1
	vsphere-csi-syncer	mirantis.azurecr.io/lcm/kubernetes/vsphere-csi-syncer:v3.0.2-1
	vsphere-vm-template-controller ^Updated	mirantis.azurecr.io/core/vsphere-vm-template-controller:1.39.28

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts	iam ^Updated	https://binary.mirantis.com/core/helm/iam-1.39.28.tgz
Docker images	kubectl	mirantis.azurecr.io/stacklight/kubectl:1.22-20240501023013
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-ba8ada4-20240405150338
	mariadb ^Updated	mirantis.azurecr.io/general/mariadb:10.6.17-focal-20240523075821
	mcc-keycloak	mirantis.azurecr.io/iam/mcc-keycloak:23.0.6-20240216125244

See also

Patch releases

2.26.4¶

The Container Cloud patch release 2.26.4, which is based on the 2.26.0 major release, provides the following updates:

Support for the patch Cluster releases 16.1.4 and 17.1.4 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 24.1.4.
Support for MKE 3.7.8.
Bare metal: update of Ubuntu mirror from 20.04~20240411171541 to 20.04~20240502102020 along with update of minor kernel version from 5.15.0-102-generic to 5.15.0-105-generic.
Security fixes for CVEs in images.
Bug fixes.

For main deliverables of the parent Container Cloud release of 2.26.4, refer to 2.26.0.

Security notes¶

The table below includes the total numbers of addressed unique and common CVEs in images by product component since the Container Cloud 2.26.3 patch release. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	Critical	High	Total
Ceph	Unique	0	1	1
Ceph	Common	0	3	3
StackLight	Unique	2	8	10
StackLight	Common	6	9	15

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 24.1.4: Security notes.

Addressed issues¶

The following issues have been addressed in the Container Cloud patch release 2.26.4 along with the patch Cluster releases 17.1.4 and 16.1.4.

[41806] [Container Cloud web UI] Fixed the issue with failure to configure management cluster using the Configure cluster web UI menu without updating the Keycloak Truststore settings.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.26.4 including the Cluster releases 17.1.4 and 16.1.4.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[46245] Lack of access permissions for HOC and HOCM objects¶

Fixed in 2.28.0 (17.3.0 and 16.3.0)

kubectl --kubeconfig ~/.kube/mgmt-config get hocm

Error from server (Forbidden): hostosconfigurationmodules.kaas.mirantis.com is forbidden:
User "2d74348b-5669-4c65-af31-6c05dbedac5f" cannot list resource "hostosconfigurationmodules"
in API group "kaas.mirantis.com" at the cluster scope: access denied

Workaround:

Modify the global-admin role by adding a new entry with the following contents to the rules list:

kubectl edit clusterroles kaas-global-admin

- apiGroups: [kaas.mirantis.com]
  resources: [hostosconfigurationmodules]
  verbs: ['*']

For each Container Cloud project, modify the kaas-operator role by adding a new entry with the following contents to the rules list:

kubectl -n <projectName> edit roles kaas-operator

- apiGroups: [kaas.mirantis.com]
  resources: [hostosconfigurations]
  verbs: ['*']

[42408] Kernel is not updated on manager nodes after cluster update¶

Fixed in 17.1.5 and 16.1.5

After managed cluster update, old versions of system packages, including kernel, may remain on the manager nodes. This issue occurs because the task responsible for updating packages fails to run after updating Ubuntu mirrors.

As a workaround, manually run apt-get upgrade on every manager node after the cluster update but before rebooting the node.

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[41305] DHCP responses are lost between dnsmasq and dhcp-relay pods¶

Fixed in 2.28.0 (17.3.0 and 16.3.0)

The issue was observed on environments having management cluster nodes configured with a single L2 segment used for all network traffic (PXE and LCM/management networks).

To verify whether the cluster is affected:

Verify whether the dnsmasq and dhcp-relay pods run on the same node in the management cluster:

kubectl -n kaas get pods -o wide| grep -e "dhcp\|dnsmasq"

Example of system response:

dhcp-relay-7d85f75f76-5vdw2   2/2   Running   2 (36h ago)   36h   10.10.0.122     kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>
dnsmasq-8f4b484b4-slhbd       5/5   Running   1 (36h ago)   36h   10.233.123.75   kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>

If this is the case, proceed to the workaround below.

Workaround:

Make sure that at least two management cluster nodes are schedulable:

kubectl get node

Example of a positive system response:

NAME                                             STATUS   ROLES    AGE   VERSION
kaas-node-bcedb87b-b3ce-46a4-a4ca-ea3068689e40   Ready    master   37h   v1.27.10-mirantis-1
kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   Ready    master   37h   v1.27.10-mirantis-1
kaas-node-ad5a6f51-b98f-43c3-91d5-55fed3d0ff21   Ready    master   37h   v1.27.10-mirantis-1

Delete the dhcp-relay pod:

kubectl -n kaas delete pod <dhcp-relay-xxxxx>

Verify that the dnsmasq and dhcp-relay pods are scheduled into different nodes:

kubectl -n kaas get pods -o wide| grep -e "dhcp\|dnsmasq"

Example of a positive system response:

dhcp-relay-7d85f75f76-rkv03   2/2   Running   0             49s   10.10.0.121     kaas-node-bcedb87b-b3ce-46a4-a4ca-ea3068689e40   <none>   <none>
dnsmasq-8f4b484b4-slhbd       5/5   Running   1 (37h ago)   37h   10.233.123.75   kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

LCM¶

[41540] LCM Agent cannot grab storage information on a host¶

Fixed in 17.1.5 and 16.1.5

Due to issues with managing physical NVME devices, lcm-agent cannot grab storage information on a host. As a result, lcmmachine.status.hostinfo.hardware is empty and the following example error is present in logs:

{"level":"error","ts":"2024-05-02T12:26:10Z","logger":"agent", \
"msg":"get hardware details", \
"host":"kaas-node-548b2861-aed0-41c9-8ff2-10c5476b000b", \
"error":"new storage info: get disk info \"nvme0c0n1\": \
invoke command: exit status 1","errorVerbose":"exit status 1

As a workaround, on the affected node, create a symlink for any device indicated in lcm-agent logs. For example:

ln -sfn /dev/nvme0n1 /dev/nvme0c0n1

[39437] Failure to replace a master node on a Container Cloud cluster¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

During the replacement of a master node on a cluster of any type, the process may get stuck with Kubelet's NodeReady condition is Unknown in the machine status on the remaining master nodes.

As a workaround, log in on the affected node and run the following command:

docker restart ucp-kubelet

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

[30294] Replacement of a master node is stuck on the calico-node Pod start¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a master node on a cluster of any type, the calico-node Pod fails to start on a new node that has the same IP address as the node being replaced.

Workaround:

From a CLI with an MKE client bundle, create a shell alias to start calicoctl using the mirantis/ucp-dsinfo image:

Since MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/key.pem \
-e ETCD_CA_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/ca.pem \
-e ETCD_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/cert.pem \
-v /var/run/calico:/var/run/calico \
-v /var/lib/docker/volumes/ucp-kv-certs/_data:/var/lib/docker/volumes/ucp-kv-certs/_data:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl \
"

Before MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
-e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
-e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
-v /var/run/calico:/var/run/calico \
-v ucp-node-certs:/ucp-node-certs:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl --allow-version-mismatch \
"

In the above command, replace the following values with the corresponding settings of the affected cluster:

<etcdEndpoint> is the etcd endpoint defined in the Calico configuration file. For example, ETCD_ENDPOINTS=127.0.0.1:12378
<mkeVersion> is the MKE version installed on your cluster. For example, mirantis/ucp-dsinfo:3.5.7.

Verify the node list on the cluster:
```
kubectl get node
```
Compare this list with the node list in Calico to identify the old node:
```
calicoctl get node -o wide
```

Remove the old node from Calico:

calicoctl delete node kaas-node-<nodeID>

[5782] Manager machine fails to be deployed during node replacement¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a manager machine, the following problems may occur:

The system adds the node to Docker swarm but not to Kubernetes
The node Deployment gets stuck with failed RethinkDB health checks

Workaround:

Delete the failed node.
Wait for the MKE cluster to become healthy. To monitor the cluster status:
1. Log in to the MKE web UI as described in Connect to the Mirantis Kubernetes Engine web UI.
2. Monitor the cluster status as described in MKE Operations Guide: Monitor an MKE cluster with the MKE web UI.
Deploy a new node.

[5568] The calico-kube-controllers Pod fails to clean up resources¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During the unsafe or forced deletion of a manager machine running the calico-kube-controllers Pod in the kube-system namespace, the following issues occur:

The calico-kube-controllers Pod fails to clean up resources associated with the deleted node
The calico-node Pod may fail to start up on a newly created node if the machine is provisioned with the same IP address as the deleted machine had

As a workaround, before deletion of the node running the calico-kube-controllers Pod, cordon and drain the node:

kubectl cordon <nodeName>
kubectl drain <nodeName>

Ceph¶

[41819] Graceful cluster reboot is blocked by the Ceph ClusterWorkloadLocks¶

Fixed in 2.27.0 (17.2.0 and 16.2.0)

During graceful reboot of a cluster with Ceph enabled, the reboot is blocked with the following message in the MiraCephMaintenance object status:

message: ClusterMaintenanceRequest found, Ceph Cluster is not ready to upgrade,
 delaying cluster maintenance

As a workaround, add the following snippet to the cephFS section under metadataServer in the spec section of <kcc-name>.yaml in the Ceph cluster:

cephClusterSpec:
  sharedFilesystem:
    cephFS:
    - name: cephfs-store
      metadataServer:
        activeCount: 1
        healthCheck:
          livenessProbe:
            probe:
              failureThreshold: 5
              initialDelaySeconds: 30
              periodSeconds: 30
              successThreshold: 1
              timeoutSeconds: 5

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

StackLight¶

[42304] Failure of shard relocation in the OpenSearch cluster¶

Fixed in 17.2.0, 16.2.0, 17.1.6, 16.1.6

{created_by_kind="StatefulSet",created_by_name="opensearch-master",namespace="stacklight"}

Caution

To verify that the initialization of shards is ongoing:

kubectl exec -it pod/opensearch-master-0 -n stacklight -c opensearch -- bash

curl "http://localhost:9200/_cat/shards" | grep INITIALIZING

Example of system response:

.ds-system-000072    2 r INITIALIZING    10.232.182.135 opensearch-master-1
.ds-system-000073    1 r INITIALIZING    10.232.7.145   opensearch-master-2
.ds-system-000073    2 r INITIALIZING    10.232.182.135 opensearch-master-1
.ds-audit-000001     2 r INITIALIZING    10.232.7.145   opensearch-master-2

You can additionally analyze the exact level of throttling and the current CPU usage on the Kubernetes Containers dashboard in Grafana.

Workaround:

Verify the currently configured CPU requests and limits for the opensearch containers:

kubectl -n stacklight get statefulset.apps/opensearch-master -o jsonpath="{.spec.template.spec.containers[?(@.name=='opensearch')].resources}"

Example of system response:

{"limits":{"cpu":"600m","memory":"8Gi"},"requests":{"cpu":"500m","memory":"6Gi"}}

In the example above, the CPU request is 500m and the CPU limit is 600m.

Increase the CPU limit to a reasonably high number.

For example, the default CPU limit for the clusters with the clusterSize:large parameter set was increased from 8000m to 12000m for StackLight in Container Cloud 2.27.0 (Cluster releases 17.2.0 and 16.2.0).

Note

For details, on the clusterSize parameter, see MOSK Operations Guide: StackLight configuration parameters - Cluster size.

If the defaults are already overridden on the affected cluster using the resourcesPerClusterSize or resources parameters as described in MOSK Operations Guide: StackLight configuration parameters - Resource limits, then the exact recommended number depends on the currently set limit.

Mirantis recommends increasing the limit by 50%. If it does not resolve the issue, another increase iteration will be required.
When you select the required CPU limit, increase it as described in MOSK Operations Guide: StackLight configuration parameters - Resource limits.

If the CPU limit for the opensearch component is already set, increase it in the Cluster object for the opensearch parameter. Otherwise, the default StackLight limit is used. In this case, increase the CPU limit for the opensearch component using the resources parameter.
Wait until all opensearch-master pods are recreated with the new CPU limits and become running and ready.

To verify the current CPU limit for every opensearch container in every opensearch-master pod separately:
```
kubectl -n stacklight get pod/opensearch-master-<podSuffixNumber> -o jsonpath="{.spec.containers[?(@.name=='opensearch')].resources}"
```
In the command above, replace <podSuffixNumber> with the name of the pod suffix. For example, pod/opensearch-master-0 or pod/opensearch-master-2.

Example of system response:
```
{"limits":{"cpu":"900m","memory":"8Gi"},"requests":{"cpu":"500m","memory":"6Gi"}}
```
The waiting time may take up to 20 minutes depending on the cluster size.

If the KubeContainersCPUThrottlingHigh alert is still firing, proceed with another iteration of the CPU limit increase.

[40020] Rollover policy update is not appllied to the current index¶

Fixed in 17.2.0, 16.2.0, 17.1.6, 16.1.6

While updating rollover_policy for the current system* and audit* data streams, the update is not applied to indices.

2024-05-31 13:16:04,459 ERROR   Failed to complete action: delete_indices.  <class 'curator.exceptions.FailedExecution'>: Exception encountered.  Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: RequestError(400, 'illegal_argument_exception', 'index [.ds-audit-000001] is the write index for data stream [audit] and cannot be deleted')

2024-05-31 13:16:04,459 ERROR   Failed to complete action: delete_indices.  <class 'curator.exceptions.FailedExecution'>: Exception encountered.  Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: RequestError(400, 'illegal_argument_exception', 'index [.ds-system-000001] is the write index for data stream [system] and cannot be deleted')

Note

Instead of .ds-audit-000001 or .ds-system-000001 index names, similar names can be present with the same prefix but different suffix numbers.

If the above mentioned alert and errors are present, an immediate action is required, because it indicates that the corresponding index size has already exceeded the space allocated for the index.

To verify that the cluster is affected:

Caution

Verify and apply the workaround to both index patterns, system and audit, separately.

If one of indices is affected, the second one is most likely affected as well. Although in rare cases, only one index may be affected.

kubectl exec -it pod/opensearch-master-0 -n stacklight -c opensearch -- bash

Verify that the rollover policy is present:
- system:
```
curl localhost:9200/_plugins/_ism/policies/system_rollover_policy
```
- audit:
```
curl localhost:9200/_plugins/_ism/policies/audit_rollover_policy
```
The cluster is affected if the rollover policy is missing. Otherwise, proceed to the following step.

Verify the system response from the previous step. For example:

{"_id":"system_rollover_policy","_version":7229,"_seq_no":42362,"_primary_term":28,"policy":{"policy_id":"system_rollover_policy","description":"system index rollover policy.","last_updated_time":1708505222430,"schema_version":19,"error_notification":null,"default_state":"rollover","states":[{"name":"rollover","actions":[{"retry":{"count":3,"backoff":"exponential","delay":"1m"},"rollover":{"min_size":"14746mb","copy_alias":false}}],"transitions":[]}],"ism_template":[{"index_patterns":["system*"],"priority":200,"last_updated_time":1708505222430}]}}

Verify and capture the following items separately for every policy:

The _seq_no and _primary_term values
The rollover policy threshold, which is defined in policy.states[0].actions[0].rollover.min_size

List indices:

system:

curl localhost:9200/_cat/indices | grep system

Example of system response:

[...]
green open .ds-system-000001   FjglnZlcTKKfKNbosaE9Aw 2 1 1998295  0   1gb 507.9mb

audit:

curl localhost:9200/_cat/indices | grep audit

Example of system response:

[...]
green open .ds-audit-000001   FjglnZlcTKKfKNbosaE9Aw 2 1 1998295  0   1gb 507.9mb

Select the index with the highest number and verify the rollover policy attached to the index:
- system:
```
curl localhost:9200/_plugins/_ism/explain/.ds-system-000001
```
- audit:
```
curl localhost:9200/_plugins/_ism/explain/.ds-audit-000001
```
- If the rollover policy is not attached, the cluster is affected.
- If the rollover policy is attached but _seq_no and _primary_term numbers do not match the previously captured ones, the cluster is affected.
- If the index size drastically exceeds the defined threshold of the rollover policy (which is the previously captured min_size), the cluster is most probably affected.

Workaround:

kubectl exec -it pod/opensearch-master-0 -n stacklight -c opensearch -- bash

If the policy is attached to the index but has different _seq_no and _primary_term, remove the policy from the index:

Note

Use the index with the highest number in the name, which was captured during verification procedure.
- system:
```
curl -XPOST localhost:9200/_plugins/_ism/remove/.ds-system-000001
```
- audit:
```
curl -XPOST localhost:9200/_plugins/_ism/remove/.ds-audit-000001
```

Re-add the policy:

system:

curl -XPOST -H "Content-type: application/json" localhost:9200/_plugins/_ism/add/system* -d'{"policy_id":"system_rollover_policy"}'

audit:

curl -XPOST -H "Content-type: application/json" localhost:9200/_plugins/_ism/add/audit* -d'{"policy_id":"audit_rollover_policy"}'

Perform again the last step of the cluster verification procedure provided above and make sure that the policy is attached to the index and has the same _seq_no and _primary_term.

If the index size drastically exceeds the defined threshold of the rollover policy (which is the previously captured min_size), wait up to 15 minutes and verify that the additional index is created with the consecutive number in the index name. For example:
- system: if you applied changes to .ds-system-000001, wait until .ds-system-000002 is created.
- audit: if you applied changes to .ds-audit-000001, wait until .ds-audit-000002 is created.
If such index is not created, escalate the issue to Mirantis support.

Update notes¶

This section describes the specific actions you as a cloud operator need to complete before or after your Container Cloud cluster update to the Cluster releases 17.1.4 or 16.1.4.

Post-update actions¶

Configure Kubernetes auditing and profiling for log rotation¶

[cluster_config]
  kube_api_server_audit_log_maxage=30
  kube_api_server_audit_log_maxbackup=10
  kube_api_server_audit_log_maxsize=10

For the configuration procedure, see MKE documentation: Configure an existing MKE cluster.

While using this procedure, replace the command to upload the newly edited MKE configuration file with the following one:

curl --silent --insecure -X PUT -H "X-UCP-Allow-Restricted-API: i-solemnly-swear-i-am-up-to-no-good" -H "accept: application/toml" -H "Authorization: Bearer $AUTHTOKEN" --upload-file 'mke-config.toml' https://$MKE_HOST/api/ucp/config-toml

The value for MKE_HOST has the <loadBalancerHost>:6443 format, where loadBalancerHost is the corresponding field in the cluster status.
The value for MKE_PASSWORD is taken from the ucp-admin-password-<clusterName> secret in the cluster namespace of the management cluster.
The value for MKE_USERNAME is always admin.

Artifacts¶

This section lists the artifacts of components included in the Container Cloud patch release 2.26.4. For artifacts of the Cluster releases introduced in 2.26.4, see patch Cluster releases 17.1.4 and 16.1.4.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries ^Updated	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20240502103738
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20240502103738
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.39.26.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.39.26.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.39.26.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.39.26.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.39.26.tgz
	local-volume-provisioner	https://binary.mirantis.com/core/helm/local-volume-provisioner-1.39.26.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.39.26.tgz
Docker images	ambasador ^Updated	mirantis.azurecr.io/core/external/nginx:1.39.26
	baremetal-dnsmasq	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-26-alpine-20240408141922
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-2-26-alpine-20240415095355
	bm-collective	mirantis.azurecr.io/bm/bm-collective:base-2-26-alpine-20240408142218
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.39.26
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:yoga-jammy-20240510100941
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:yoga-jammy-20240510100941
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20240117102150
	kaas-ipam	mirantis.azurecr.io/bm/kaas-ipam:base-2-26-alpine-20240408150853
	kubernetes-entrypoint ^Updated	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-ba8ada4-20240405150338
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.14-focal-20240311120505
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.24.0-47-gf77368e
	metallb-controller	mirantis.azurecr.io/bm/metallb/controller:v0.13.12-ef4c9453-amd64
	metallb-speaker	mirantis.azurecr.io/bm/metallb/speaker:v0.13.12-ef4c9453-amd64
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20240129163811

Core artifacts¶

Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.39.26.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.39.26.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.39.26.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.39.26.tgz
	byo-credentials-controller	https://binary.mirantis.com/core/helm/byo-credentials-controller-1.39.26.tgz
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.39.26.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.39.26.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.39.26.tgz
	cinder-csi-plugin	https://binary.mirantis.com/core/helm/cinder-csi-plugin-1.39.26.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.39.26.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.39.26.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.39.26.tgz
	host-os-modules-controller	https://binary.mirantis.com/core/helm/host-os-modules-controller-1.39.26.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.39.26.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.39.26.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.39.26.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.39.26.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.39.26.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.39.26.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.39.26.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.39.26.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.39.26.tgz
	metrics-server	https://binary.mirantis.com/core/helm/metrics-server-1.39.26.tgz
	openstack-cloud-controller-manager	https://binary.mirantis.com/core/helm/openstack-cloud-controller-manager-1.39.26.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.39.26.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.39.26.tgz
	policy-controller	https://binary.mirantis.com/core/helm/policy-controller-1.39.26.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.39.26.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.39.26.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.39.26.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.39.26.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.39.26.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.39.26.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.39.26.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.39.26.tgz
	vsphere-cloud-controller-manager	https://binary.mirantis.com/core/helm/vsphere-cloud-controller-manager-1.39.26.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.39.26.tgz
	vsphere-csi-plugin	https://binary.mirantis.com/core/helm/vsphere-csi-plugin-1.39.26.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.39.26.tgz
	vsphere-vm-template-controller	https://binary.mirantis.com/core/helm/vsphere-vm-template-controller-1.39.26.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.39.26
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.39.26
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.39.26
	byo-credentials-controller ^Updated	mirantis.azurecr.io/core/byo-credentials-controller:1.39.26
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.39.26
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-6
	cinder-csi-plugin ^Updated	mirantis.azurecr.io/lcm/kubernetes/cinder-csi-plugin:v1.27.2-16
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.39.26
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.39.26
	csi-attacher	mirantis.azurecr.io/lcm/k8scsi/csi-attacher:v4.2.0-5
	csi-node-driver-registrar	mirantis.azurecr.io/lcm/k8scsi/csi-node-driver-registrar:v2.7.0-5
	csi-provisioner	mirantis.azurecr.io/lcm/k8scsi/csi-provisioner:v3.4.1-5
	csi-resizer	mirantis.azurecr.io/lcm/k8scsi/csi-resizer:v1.7.0-5
	csi-snapshotter	mirantis.azurecr.io/lcm/k8scsi/csi-snapshotter:v6.2.1-mcc-4
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.39.26
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.39.26
	host-os-modules-controller ^Updated	mirantis.azurecr.io/core/host-os-modules-controller:1.39.26
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.39.26
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.39.26
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.39.26
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.39.26
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.39.26
	livenessprobe	mirantis.azurecr.io/lcm/k8scsi/livenessprobe:v2.9.0-5
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.39.26
	mcc-haproxy	mirantis.azurecr.io/lcm/mcc-haproxy:v0.24.0-47-gf77368e
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.24.0-47-gf77368e
	metrics-server	mirantis.azurecr.io/lcm/metrics-server-amd64:v0.6.3-7
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.39.26
	openstack-cloud-controller-manager ^Updated	mirantis.azurecr.io/lcm/kubernetes/openstack-cloud-controller-manager:v1.27.2-16
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.39.26
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.39.26
	policy-controller ^Updated	mirantis.azurecr.io/core/policy-controller:1.39.26
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.39.26
	proxy-controller ^Updated	mirantis.azurecr.io/core/proxy-controller:1.39.26
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.39.26
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-9
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.39.26
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.39.26
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.39.26
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-10-g24a0d69
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.39.26
	vsphere-cloud-controller-manager	mirantis.azurecr.io/lcm/kubernetes/vsphere-cloud-controller-manager:v1.27.0-6
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.39.26
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.39.26
	vsphere-csi-driver	mirantis.azurecr.io/lcm/kubernetes/vsphere-csi-driver:v3.0.2-1
	vsphere-csi-syncer	mirantis.azurecr.io/lcm/kubernetes/vsphere-csi-syncer:v3.0.2-1
	vsphere-vm-template-controller ^Updated	mirantis.azurecr.io/core/vsphere-vm-template-controller:1.39.26

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts	iam ^Updated	https://binary.mirantis.com/core/helm/iam-1.39.26.tgz
Docker images	kubectl ^Updated	mirantis.azurecr.io/stacklight/kubectl:1.22-20240501023013
	kubernetes-entrypoint ^Updated	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-ba8ada4-20240405150338
	mariadb ^Updated	mirantis.azurecr.io/general/mariadb:10.6.17-focal-20240327104027
	mcc-keycloak	mirantis.azurecr.io/iam/mcc-keycloak:23.0.6-20240216125244

See also

Patch releases

2.26.3¶

The Container Cloud patch release 2.26.3, which is based on the 2.26.0 major release, provides the following updates:

Support for the patch Cluster releases 16.1.3 and 17.1.3 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 24.1.3.
Support for MKE 3.7.7.
Bare metal: update of Ubuntu mirror from 20.04~20240324172903 to 20.04~20240411171541 along with update of minor kernel version from 5.15.0-101-generic to 5.15.0-102-generic.
Security fixes for CVEs in images.
Bug fixes.

For main deliverables of the parent Container Cloud release of 2.26.3, refer to 2.26.0.

Security notes¶

The table below includes the total numbers of addressed unique and common CVEs in images by product component since the Container Cloud 2.26.2 patch release. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	Critical	High	Total
Ceph	Unique	0	1	1
Ceph	Common	0	10	10
Core	Unique	0	4	4
Core	Common	0	105	105
StackLight	Unique	1	4	5
StackLight	Common	1	24	25

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 24.1.3: Security notes.

Addressed issues¶

The following issues have been addressed in the Container Cloud patch release 2.26.3 along with the patch Cluster releases 17.1.3 and 16.1.3.

[40811] [LCM] Fixed the issue with the DaemonSet Pod remaining on the deleted node in the Terminating state during machine deletion.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.26.3 including the Cluster releases 17.1.3 and 16.1.3.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[46245] Lack of access permissions for HOC and HOCM objects¶

Fixed in 2.28.0 (17.3.0 and 16.3.0)

kubectl --kubeconfig ~/.kube/mgmt-config get hocm

Error from server (Forbidden): hostosconfigurationmodules.kaas.mirantis.com is forbidden:
User "2d74348b-5669-4c65-af31-6c05dbedac5f" cannot list resource "hostosconfigurationmodules"
in API group "kaas.mirantis.com" at the cluster scope: access denied

Workaround:

Modify the global-admin role by adding a new entry with the following contents to the rules list:

kubectl edit clusterroles kaas-global-admin

- apiGroups: [kaas.mirantis.com]
  resources: [hostosconfigurationmodules]
  verbs: ['*']

For each Container Cloud project, modify the kaas-operator role by adding a new entry with the following contents to the rules list:

kubectl -n <projectName> edit roles kaas-operator

- apiGroups: [kaas.mirantis.com]
  resources: [hostosconfigurations]
  verbs: ['*']

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[41305] DHCP responses are lost between dnsmasq and dhcp-relay pods¶

Fixed in 2.28.0 (17.3.0 and 16.3.0)

The issue was observed on environments having management cluster nodes configured with a single L2 segment used for all network traffic (PXE and LCM/management networks).

To verify whether the cluster is affected:

Verify whether the dnsmasq and dhcp-relay pods run on the same node in the management cluster:

kubectl -n kaas get pods -o wide| grep -e "dhcp\|dnsmasq"

Example of system response:

dhcp-relay-7d85f75f76-5vdw2   2/2   Running   2 (36h ago)   36h   10.10.0.122     kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>
dnsmasq-8f4b484b4-slhbd       5/5   Running   1 (36h ago)   36h   10.233.123.75   kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>

If this is the case, proceed to the workaround below.

Workaround:

Make sure that at least two management cluster nodes are schedulable:

kubectl get node

Example of a positive system response:

NAME                                             STATUS   ROLES    AGE   VERSION
kaas-node-bcedb87b-b3ce-46a4-a4ca-ea3068689e40   Ready    master   37h   v1.27.10-mirantis-1
kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   Ready    master   37h   v1.27.10-mirantis-1
kaas-node-ad5a6f51-b98f-43c3-91d5-55fed3d0ff21   Ready    master   37h   v1.27.10-mirantis-1

Delete the dhcp-relay pod:

kubectl -n kaas delete pod <dhcp-relay-xxxxx>

Verify that the dnsmasq and dhcp-relay pods are scheduled into different nodes:

kubectl -n kaas get pods -o wide| grep -e "dhcp\|dnsmasq"

Example of a positive system response:

dhcp-relay-7d85f75f76-rkv03   2/2   Running   0             49s   10.10.0.121     kaas-node-bcedb87b-b3ce-46a4-a4ca-ea3068689e40   <none>   <none>
dnsmasq-8f4b484b4-slhbd       5/5   Running   1 (37h ago)   37h   10.233.123.75   kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

LCM¶

[41540] LCM Agent cannot grab storage information on a host¶

Fixed in 17.1.5 and 16.1.5

{"level":"error","ts":"2024-05-02T12:26:10Z","logger":"agent", \
"msg":"get hardware details", \
"host":"kaas-node-548b2861-aed0-41c9-8ff2-10c5476b000b", \
"error":"new storage info: get disk info \"nvme0c0n1\": \
invoke command: exit status 1","errorVerbose":"exit status 1

As a workaround, on the affected node, create a symlink for any device indicated in lcm-agent logs. For example:

ln -sfn /dev/nvme0n1 /dev/nvme0c0n1

[39437] Failure to replace a master node on a Container Cloud cluster¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

During the replacement of a master node on a cluster of any type, the process may get stuck with Kubelet's NodeReady condition is Unknown in the machine status on the remaining master nodes.

As a workaround, log in on the affected node and run the following command:

docker restart ucp-kubelet

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

[30294] Replacement of a master node is stuck on the calico-node Pod start¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a master node on a cluster of any type, the calico-node Pod fails to start on a new node that has the same IP address as the node being replaced.

Workaround:

From a CLI with an MKE client bundle, create a shell alias to start calicoctl using the mirantis/ucp-dsinfo image:

Since MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/key.pem \
-e ETCD_CA_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/ca.pem \
-e ETCD_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/cert.pem \
-v /var/run/calico:/var/run/calico \
-v /var/lib/docker/volumes/ucp-kv-certs/_data:/var/lib/docker/volumes/ucp-kv-certs/_data:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl \
"

Before MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
-e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
-e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
-v /var/run/calico:/var/run/calico \
-v ucp-node-certs:/ucp-node-certs:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl --allow-version-mismatch \
"

In the above command, replace the following values with the corresponding settings of the affected cluster:

<etcdEndpoint> is the etcd endpoint defined in the Calico configuration file. For example, ETCD_ENDPOINTS=127.0.0.1:12378
<mkeVersion> is the MKE version installed on your cluster. For example, mirantis/ucp-dsinfo:3.5.7.

Verify the node list on the cluster:
```
kubectl get node
```
Compare this list with the node list in Calico to identify the old node:
```
calicoctl get node -o wide
```

Remove the old node from Calico:

calicoctl delete node kaas-node-<nodeID>

[5782] Manager machine fails to be deployed during node replacement¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a manager machine, the following problems may occur:

The system adds the node to Docker swarm but not to Kubernetes
The node Deployment gets stuck with failed RethinkDB health checks

Workaround:

Delete the failed node.
Wait for the MKE cluster to become healthy. To monitor the cluster status:
1. Log in to the MKE web UI as described in Connect to the Mirantis Kubernetes Engine web UI.
2. Monitor the cluster status as described in MKE Operations Guide: Monitor an MKE cluster with the MKE web UI.
Deploy a new node.

[5568] The calico-kube-controllers Pod fails to clean up resources¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During the unsafe or forced deletion of a manager machine running the calico-kube-controllers Pod in the kube-system namespace, the following issues occur:

The calico-kube-controllers Pod fails to clean up resources associated with the deleted node
The calico-node Pod may fail to start up on a newly created node if the machine is provisioned with the same IP address as the deleted machine had

As a workaround, before deletion of the node running the calico-kube-controllers Pod, cordon and drain the node:

kubectl cordon <nodeName>
kubectl drain <nodeName>

Ceph¶

[41819] Graceful cluster reboot is blocked by the Ceph ClusterWorkloadLocks¶

Fixed in 2.27.0 (17.2.0 and 16.2.0)

During graceful reboot of a cluster with Ceph enabled, the reboot is blocked with the following message in the MiraCephMaintenance object status:

message: ClusterMaintenanceRequest found, Ceph Cluster is not ready to upgrade,
 delaying cluster maintenance

As a workaround, add the following snippet to the cephFS section under metadataServer in the spec section of <kcc-name>.yaml in the Ceph cluster:

cephClusterSpec:
  sharedFilesystem:
    cephFS:
    - name: cephfs-store
      metadataServer:
        activeCount: 1
        healthCheck:
          livenessProbe:
            probe:
              failureThreshold: 5
              initialDelaySeconds: 30
              periodSeconds: 30
              successThreshold: 1
              timeoutSeconds: 5

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

StackLight¶

[42304] Failure of shard relocation in the OpenSearch cluster¶

Fixed in 17.2.0, 16.2.0, 17.1.6, 16.1.6

{created_by_kind="StatefulSet",created_by_name="opensearch-master",namespace="stacklight"}

Caution

To verify that the initialization of shards is ongoing:

kubectl exec -it pod/opensearch-master-0 -n stacklight -c opensearch -- bash

curl "http://localhost:9200/_cat/shards" | grep INITIALIZING

Example of system response:

.ds-system-000072    2 r INITIALIZING    10.232.182.135 opensearch-master-1
.ds-system-000073    1 r INITIALIZING    10.232.7.145   opensearch-master-2
.ds-system-000073    2 r INITIALIZING    10.232.182.135 opensearch-master-1
.ds-audit-000001     2 r INITIALIZING    10.232.7.145   opensearch-master-2

You can additionally analyze the exact level of throttling and the current CPU usage on the Kubernetes Containers dashboard in Grafana.

Workaround:

Verify the currently configured CPU requests and limits for the opensearch containers:

kubectl -n stacklight get statefulset.apps/opensearch-master -o jsonpath="{.spec.template.spec.containers[?(@.name=='opensearch')].resources}"

Example of system response:

{"limits":{"cpu":"600m","memory":"8Gi"},"requests":{"cpu":"500m","memory":"6Gi"}}

In the example above, the CPU request is 500m and the CPU limit is 600m.

Increase the CPU limit to a reasonably high number.

For example, the default CPU limit for the clusters with the clusterSize:large parameter set was increased from 8000m to 12000m for StackLight in Container Cloud 2.27.0 (Cluster releases 17.2.0 and 16.2.0).

Note

For details, on the clusterSize parameter, see MOSK Operations Guide: StackLight configuration parameters - Cluster size.

If the defaults are already overridden on the affected cluster using the resourcesPerClusterSize or resources parameters as described in MOSK Operations Guide: StackLight configuration parameters - Resource limits, then the exact recommended number depends on the currently set limit.

Mirantis recommends increasing the limit by 50%. If it does not resolve the issue, another increase iteration will be required.
When you select the required CPU limit, increase it as described in MOSK Operations Guide: StackLight configuration parameters - Resource limits.

If the CPU limit for the opensearch component is already set, increase it in the Cluster object for the opensearch parameter. Otherwise, the default StackLight limit is used. In this case, increase the CPU limit for the opensearch component using the resources parameter.
Wait until all opensearch-master pods are recreated with the new CPU limits and become running and ready.

To verify the current CPU limit for every opensearch container in every opensearch-master pod separately:
```
kubectl -n stacklight get pod/opensearch-master-<podSuffixNumber> -o jsonpath="{.spec.containers[?(@.name=='opensearch')].resources}"
```
In the command above, replace <podSuffixNumber> with the name of the pod suffix. For example, pod/opensearch-master-0 or pod/opensearch-master-2.

Example of system response:
```
{"limits":{"cpu":"900m","memory":"8Gi"},"requests":{"cpu":"500m","memory":"6Gi"}}
```
The waiting time may take up to 20 minutes depending on the cluster size.

If the KubeContainersCPUThrottlingHigh alert is still firing, proceed with another iteration of the CPU limit increase.

[40020] Rollover policy update is not appllied to the current index¶

Fixed in 17.2.0, 16.2.0, 17.1.6, 16.1.6

While updating rollover_policy for the current system* and audit* data streams, the update is not applied to indices.

2024-05-31 13:16:04,459 ERROR   Failed to complete action: delete_indices.  <class 'curator.exceptions.FailedExecution'>: Exception encountered.  Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: RequestError(400, 'illegal_argument_exception', 'index [.ds-audit-000001] is the write index for data stream [audit] and cannot be deleted')

2024-05-31 13:16:04,459 ERROR   Failed to complete action: delete_indices.  <class 'curator.exceptions.FailedExecution'>: Exception encountered.  Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: RequestError(400, 'illegal_argument_exception', 'index [.ds-system-000001] is the write index for data stream [system] and cannot be deleted')

Note

Instead of .ds-audit-000001 or .ds-system-000001 index names, similar names can be present with the same prefix but different suffix numbers.

If the above mentioned alert and errors are present, an immediate action is required, because it indicates that the corresponding index size has already exceeded the space allocated for the index.

To verify that the cluster is affected:

Caution

Verify and apply the workaround to both index patterns, system and audit, separately.

If one of indices is affected, the second one is most likely affected as well. Although in rare cases, only one index may be affected.

kubectl exec -it pod/opensearch-master-0 -n stacklight -c opensearch -- bash

Verify that the rollover policy is present:
- system:
```
curl localhost:9200/_plugins/_ism/policies/system_rollover_policy
```
- audit:
```
curl localhost:9200/_plugins/_ism/policies/audit_rollover_policy
```
The cluster is affected if the rollover policy is missing. Otherwise, proceed to the following step.

Verify the system response from the previous step. For example:

{"_id":"system_rollover_policy","_version":7229,"_seq_no":42362,"_primary_term":28,"policy":{"policy_id":"system_rollover_policy","description":"system index rollover policy.","last_updated_time":1708505222430,"schema_version":19,"error_notification":null,"default_state":"rollover","states":[{"name":"rollover","actions":[{"retry":{"count":3,"backoff":"exponential","delay":"1m"},"rollover":{"min_size":"14746mb","copy_alias":false}}],"transitions":[]}],"ism_template":[{"index_patterns":["system*"],"priority":200,"last_updated_time":1708505222430}]}}

Verify and capture the following items separately for every policy:

The _seq_no and _primary_term values
The rollover policy threshold, which is defined in policy.states[0].actions[0].rollover.min_size

List indices:

system:

curl localhost:9200/_cat/indices | grep system

Example of system response:

[...]
green open .ds-system-000001   FjglnZlcTKKfKNbosaE9Aw 2 1 1998295  0   1gb 507.9mb

audit:

curl localhost:9200/_cat/indices | grep audit

Example of system response:

[...]
green open .ds-audit-000001   FjglnZlcTKKfKNbosaE9Aw 2 1 1998295  0   1gb 507.9mb

Select the index with the highest number and verify the rollover policy attached to the index:
- system:
```
curl localhost:9200/_plugins/_ism/explain/.ds-system-000001
```
- audit:
```
curl localhost:9200/_plugins/_ism/explain/.ds-audit-000001
```
- If the rollover policy is not attached, the cluster is affected.
- If the rollover policy is attached but _seq_no and _primary_term numbers do not match the previously captured ones, the cluster is affected.
- If the index size drastically exceeds the defined threshold of the rollover policy (which is the previously captured min_size), the cluster is most probably affected.

Workaround:

kubectl exec -it pod/opensearch-master-0 -n stacklight -c opensearch -- bash

If the policy is attached to the index but has different _seq_no and _primary_term, remove the policy from the index:

Note

Use the index with the highest number in the name, which was captured during verification procedure.
- system:
```
curl -XPOST localhost:9200/_plugins/_ism/remove/.ds-system-000001
```
- audit:
```
curl -XPOST localhost:9200/_plugins/_ism/remove/.ds-audit-000001
```

Re-add the policy:

system:

curl -XPOST -H "Content-type: application/json" localhost:9200/_plugins/_ism/add/system* -d'{"policy_id":"system_rollover_policy"}'

audit:

curl -XPOST -H "Content-type: application/json" localhost:9200/_plugins/_ism/add/audit* -d'{"policy_id":"audit_rollover_policy"}'

Perform again the last step of the cluster verification procedure provided above and make sure that the policy is attached to the index and has the same _seq_no and _primary_term.

If the index size drastically exceeds the defined threshold of the rollover policy (which is the previously captured min_size), wait up to 15 minutes and verify that the additional index is created with the consecutive number in the index name. For example:
- system: if you applied changes to .ds-system-000001, wait until .ds-system-000002 is created.
- audit: if you applied changes to .ds-audit-000001, wait until .ds-audit-000002 is created.
If such index is not created, escalate the issue to Mirantis support.

Container Cloud web UI¶

[41806] Configuration of a management cluster fails without Keycloak settings¶

Fixed in 17.1.4 and 16.1.4

During configuration of a management cluster settings using the Configure cluster web UI menu, updating the Keycloak Truststore settings is mandatory, despite being optional.

As a workaround, update the management cluster using the API or CLI.

Artifacts¶

This section lists the artifacts of components included in the Container Cloud patch release 2.26.3. For artifacts of the Cluster releases introduced in 2.26.3, see patch Cluster releases 17.1.3 and 16.1.3.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries ^Updated	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20240411174919
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20240411174919
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.39.23.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.39.23.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.39.23.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.39.23.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.39.23.tgz
	local-volume-provisioner	https://binary.mirantis.com/core/helm/local-volume-provisioner-1.39.23.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.39.23.tgz
Docker images	ambasador ^Updated	mirantis.azurecr.io/core/external/nginx:1.39.23
	baremetal-dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-26-alpine-20240408141922
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-2-26-alpine-20240408141703
	bm-collective ^Updated	mirantis.azurecr.io/bm/bm-collective:base-2-26-alpine-20240408142218
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.39.23
	ironic	mirantis.azurecr.io/openstack/ironic:yoga-jammy-20240226060024
	ironic-inspector	mirantis.azurecr.io/openstack/ironic-inspector:yoga-jammy-20240226060024
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20240117102150
	kaas-ipam ^Updated	mirantis.azurecr.io/bm/kaas-ipam:base-2-26-alpine-20240408150853
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-55b02f7-20231019172556
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.14-focal-20240311120505
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.24.0-47-gf77368e
	metallb-controller ^Updated	mirantis.azurecr.io/bm/metallb/controller:v0.13.12-ef4c9453-amd64
	metallb-speaker ^Updated	mirantis.azurecr.io/bm/metallb/speaker:v0.13.12-ef4c9453-amd64
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20240129163811

Core artifacts¶

Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.39.23.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.39.23.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.39.23.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.39.23.tgz
	byo-credentials-controller	https://binary.mirantis.com/core/helm/byo-credentials-controller-1.39.23.tgz
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.39.23.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.39.23.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.39.23.tgz
	cinder-csi-plugin	https://binary.mirantis.com/core/helm/cinder-csi-plugin-1.39.23.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.39.23.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.39.23.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.39.23.tgz
	host-os-modules-controller	https://binary.mirantis.com/core/helm/host-os-modules-controller-1.39.23.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.39.23.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.39.23.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.39.23.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.39.23.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.39.23.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.39.23.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.39.23.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.39.23.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.39.23.tgz
	metrics-server	https://binary.mirantis.com/core/helm/metrics-server-1.39.23.tgz
	openstack-cloud-controller-manager	https://binary.mirantis.com/core/helm/openstack-cloud-controller-manager-1.39.23.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.39.23.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.39.23.tgz
	policy-controller	https://binary.mirantis.com/core/helm/policy-controller-1.39.23.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.39.23.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.39.23.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.39.23.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.39.23.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.39.23.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.39.23.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.39.23.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.39.23.tgz
	vsphere-cloud-controller-manager	https://binary.mirantis.com/core/helm/vsphere-cloud-controller-manager-1.39.23.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.39.23.tgz
	vsphere-csi-plugin	https://binary.mirantis.com/core/helm/vsphere-csi-plugin-1.39.23.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.39.23.tgz
	vsphere-vm-template-controller	https://binary.mirantis.com/core/helm/vsphere-vm-template-controller-1.39.23.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.39.23
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.39.23
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.39.23
	byo-credentials-controller ^Updated	mirantis.azurecr.io/core/byo-credentials-controller:1.39.23
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.39.23
	cert-manager-controller ^Updated	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-6
	cinder-csi-plugin ^Updated	mirantis.azurecr.io/lcm/kubernetes/cinder-csi-plugin:v1.27.2-14
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.39.23
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.39.23
	csi-attacher ^Updated	mirantis.azurecr.io/lcm/k8scsi/csi-attacher:v4.2.0-5
	csi-node-driver-registrar ^Updated	mirantis.azurecr.io/lcm/k8scsi/csi-node-driver-registrar:v2.7.0-5
	csi-provisioner ^Updated	mirantis.azurecr.io/lcm/k8scsi/csi-provisioner:v3.4.1-5
	csi-resizer ^Updated	mirantis.azurecr.io/lcm/k8scsi/csi-resizer:v1.7.0-5
	csi-snapshotter ^Updated	mirantis.azurecr.io/lcm/k8scsi/csi-snapshotter:v6.2.1-mcc-4
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.39.23
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.39.23
	host-os-modules-controller ^Updated	mirantis.azurecr.io/core/host-os-modules-controller:1.39.23
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.39.23
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.39.23
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.39.23
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.39.23
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.39.23
	livenessprobe ^Updated	mirantis.azurecr.io/lcm/k8scsi/livenessprobe:v2.9.0-5
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.39.23
	mcc-haproxy	mirantis.azurecr.io/lcm/mcc-haproxy:v0.24.0-47-gf77368e
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.24.0-47-gf77368e
	metrics-server ^Updated	mirantis.azurecr.io/lcm/metrics-server-amd64:v0.6.3-7
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.39.23
	openstack-cloud-controller-manager ^Updated	mirantis.azurecr.io/lcm/kubernetes/openstack-cloud-controller-manager:v1.27.2-14
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.39.23
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.39.23
	policy-controller ^Updated	mirantis.azurecr.io/core/policy-controller:1.39.23
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.39.23
	proxy-controller ^Updated	mirantis.azurecr.io/core/proxy-controller:1.39.23
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.39.23
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-9
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.39.23
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.39.23
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.39.23
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-10-g24a0d69
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.39.23
	vsphere-cloud-controller-manager ^Updated	mirantis.azurecr.io/lcm/kubernetes/vsphere-cloud-controller-manager:v1.27.0-6
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.39.23
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.39.23
	vsphere-csi-driver	mirantis.azurecr.io/lcm/kubernetes/vsphere-csi-driver:v3.0.2-1
	vsphere-csi-syncer	mirantis.azurecr.io/lcm/kubernetes/vsphere-csi-syncer:v3.0.2-1
	vsphere-vm-template-controller ^Updated	mirantis.azurecr.io/core/vsphere-vm-template-controller:1.39.23

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts	iam ^Updated	https://binary.mirantis.com/core/helm/iam-1.39.23.tgz
Docker images	kubectl	mirantis.azurecr.io/stacklight/kubectl:1.22-20240221023016
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-55b02f7-20231019172556
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.14-focal-20240311120505
	mcc-keycloak	mirantis.azurecr.io/iam/mcc-keycloak:23.0.6-20240216125244

See also

Patch releases

2.26.2¶

The Container Cloud patch release 2.26.2, which is based on the 2.26.0 major release, provides the following updates:

Support for the patch Cluster releases 16.1.2 and 17.1.2 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 24.1.2.
Support for MKE 3.7.6.
Support for docker-ee-cli 23.0.10 in MCR 23.0.9 to fix several CVEs.
Bare metal: update of Ubuntu mirror from 20.04~20240302175618 to 20.04~20240324172903 along with update of minor kernel version from 5.15.0-97-generic to 5.15.0-101-generic.
Security fixes for CVEs in images.

For main deliverables of the parent Container Cloud release of 2.26.2, refer to 2.26.0.

Security notes¶

The table below includes the total numbers of addressed unique and common CVEs in images by product component since the Container Cloud 2.26.1 patch release. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	Critical	High	Total
Ceph	Unique	0	3	3
Ceph	Common	0	12	12
Kaas core	Unique	1	6	7
Kaas core	Common	1	11	12
StackLight	Unique	0	1	1
StackLight	Common	0	10	10

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 24.1.2: Security notes.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.26.2 including the Cluster releases 17.1.2 and 16.1.2.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[46245] Lack of access permissions for HOC and HOCM objects¶

Fixed in 2.28.0 (17.3.0 and 16.3.0)

kubectl --kubeconfig ~/.kube/mgmt-config get hocm

Error from server (Forbidden): hostosconfigurationmodules.kaas.mirantis.com is forbidden:
User "2d74348b-5669-4c65-af31-6c05dbedac5f" cannot list resource "hostosconfigurationmodules"
in API group "kaas.mirantis.com" at the cluster scope: access denied

Workaround:

Modify the global-admin role by adding a new entry with the following contents to the rules list:

kubectl edit clusterroles kaas-global-admin

- apiGroups: [kaas.mirantis.com]
  resources: [hostosconfigurationmodules]
  verbs: ['*']

For each Container Cloud project, modify the kaas-operator role by adding a new entry with the following contents to the rules list:

kubectl -n <projectName> edit roles kaas-operator

- apiGroups: [kaas.mirantis.com]
  resources: [hostosconfigurations]
  verbs: ['*']

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[41305] DHCP responses are lost between dnsmasq and dhcp-relay pods¶

Fixed in 2.28.0 (17.3.0 and 16.3.0)

The issue was observed on environments having management cluster nodes configured with a single L2 segment used for all network traffic (PXE and LCM/management networks).

To verify whether the cluster is affected:

Verify whether the dnsmasq and dhcp-relay pods run on the same node in the management cluster:

kubectl -n kaas get pods -o wide| grep -e "dhcp\|dnsmasq"

Example of system response:

dhcp-relay-7d85f75f76-5vdw2   2/2   Running   2 (36h ago)   36h   10.10.0.122     kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>
dnsmasq-8f4b484b4-slhbd       5/5   Running   1 (36h ago)   36h   10.233.123.75   kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>

If this is the case, proceed to the workaround below.

Workaround:

Make sure that at least two management cluster nodes are schedulable:

kubectl get node

Example of a positive system response:

NAME                                             STATUS   ROLES    AGE   VERSION
kaas-node-bcedb87b-b3ce-46a4-a4ca-ea3068689e40   Ready    master   37h   v1.27.10-mirantis-1
kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   Ready    master   37h   v1.27.10-mirantis-1
kaas-node-ad5a6f51-b98f-43c3-91d5-55fed3d0ff21   Ready    master   37h   v1.27.10-mirantis-1

Delete the dhcp-relay pod:

kubectl -n kaas delete pod <dhcp-relay-xxxxx>

Verify that the dnsmasq and dhcp-relay pods are scheduled into different nodes:

kubectl -n kaas get pods -o wide| grep -e "dhcp\|dnsmasq"

Example of a positive system response:

dhcp-relay-7d85f75f76-rkv03   2/2   Running   0             49s   10.10.0.121     kaas-node-bcedb87b-b3ce-46a4-a4ca-ea3068689e40   <none>   <none>
dnsmasq-8f4b484b4-slhbd       5/5   Running   1 (37h ago)   37h   10.233.123.75   kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

LCM¶

[41540] LCM Agent cannot grab storage information on a host¶

Fixed in 17.1.5 and 16.1.5

{"level":"error","ts":"2024-05-02T12:26:10Z","logger":"agent", \
"msg":"get hardware details", \
"host":"kaas-node-548b2861-aed0-41c9-8ff2-10c5476b000b", \
"error":"new storage info: get disk info \"nvme0c0n1\": \
invoke command: exit status 1","errorVerbose":"exit status 1

As a workaround, on the affected node, create a symlink for any device indicated in lcm-agent logs. For example:

ln -sfn /dev/nvme0n1 /dev/nvme0c0n1

[40811] Pod is stuck in the Terminating state on the deleted node¶

Fixed in 17.1.3 and 16.1.3

During deletion of a machine, the related DaemonSet Pod can remain on the deleted node in the Terminating state. As a workaround, manually delete the Pod:

kubectl delete pod -n <podNamespace> <podName>

[39437] Failure to replace a master node on a Container Cloud cluster¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

During the replacement of a master node on a cluster of any type, the process may get stuck with Kubelet's NodeReady condition is Unknown in the machine status on the remaining master nodes.

As a workaround, log in on the affected node and run the following command:

docker restart ucp-kubelet

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

[30294] Replacement of a master node is stuck on the calico-node Pod start¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a master node on a cluster of any type, the calico-node Pod fails to start on a new node that has the same IP address as the node being replaced.

Workaround:

From a CLI with an MKE client bundle, create a shell alias to start calicoctl using the mirantis/ucp-dsinfo image:

Since MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/key.pem \
-e ETCD_CA_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/ca.pem \
-e ETCD_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/cert.pem \
-v /var/run/calico:/var/run/calico \
-v /var/lib/docker/volumes/ucp-kv-certs/_data:/var/lib/docker/volumes/ucp-kv-certs/_data:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl \
"

Before MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
-e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
-e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
-v /var/run/calico:/var/run/calico \
-v ucp-node-certs:/ucp-node-certs:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl --allow-version-mismatch \
"

In the above command, replace the following values with the corresponding settings of the affected cluster:

<etcdEndpoint> is the etcd endpoint defined in the Calico configuration file. For example, ETCD_ENDPOINTS=127.0.0.1:12378
<mkeVersion> is the MKE version installed on your cluster. For example, mirantis/ucp-dsinfo:3.5.7.

Verify the node list on the cluster:
```
kubectl get node
```
Compare this list with the node list in Calico to identify the old node:
```
calicoctl get node -o wide
```

Remove the old node from Calico:

calicoctl delete node kaas-node-<nodeID>

[5782] Manager machine fails to be deployed during node replacement¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a manager machine, the following problems may occur:

The system adds the node to Docker swarm but not to Kubernetes
The node Deployment gets stuck with failed RethinkDB health checks

Workaround:

Delete the failed node.
Wait for the MKE cluster to become healthy. To monitor the cluster status:
1. Log in to the MKE web UI as described in Connect to the Mirantis Kubernetes Engine web UI.
2. Monitor the cluster status as described in MKE Operations Guide: Monitor an MKE cluster with the MKE web UI.
Deploy a new node.

[5568] The calico-kube-controllers Pod fails to clean up resources¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During the unsafe or forced deletion of a manager machine running the calico-kube-controllers Pod in the kube-system namespace, the following issues occur:

The calico-kube-controllers Pod fails to clean up resources associated with the deleted node
The calico-node Pod may fail to start up on a newly created node if the machine is provisioned with the same IP address as the deleted machine had

As a workaround, before deletion of the node running the calico-kube-controllers Pod, cordon and drain the node:

kubectl cordon <nodeName>
kubectl drain <nodeName>

Ceph¶

[41819] Graceful cluster reboot is blocked by the Ceph ClusterWorkloadLocks¶

Fixed in 2.27.0 (17.2.0 and 16.2.0)

During graceful reboot of a cluster with Ceph enabled, the reboot is blocked with the following message in the MiraCephMaintenance object status:

message: ClusterMaintenanceRequest found, Ceph Cluster is not ready to upgrade,
 delaying cluster maintenance

As a workaround, add the following snippet to the cephFS section under metadataServer in the spec section of <kcc-name>.yaml in the Ceph cluster:

cephClusterSpec:
  sharedFilesystem:
    cephFS:
    - name: cephfs-store
      metadataServer:
        activeCount: 1
        healthCheck:
          livenessProbe:
            probe:
              failureThreshold: 5
              initialDelaySeconds: 30
              periodSeconds: 30
              successThreshold: 1
              timeoutSeconds: 5

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

StackLight¶

[42304] Failure of shard relocation in the OpenSearch cluster¶

Fixed in 17.2.0, 16.2.0, 17.1.6, 16.1.6

{created_by_kind="StatefulSet",created_by_name="opensearch-master",namespace="stacklight"}

Caution

To verify that the initialization of shards is ongoing:

kubectl exec -it pod/opensearch-master-0 -n stacklight -c opensearch -- bash

curl "http://localhost:9200/_cat/shards" | grep INITIALIZING

Example of system response:

.ds-system-000072    2 r INITIALIZING    10.232.182.135 opensearch-master-1
.ds-system-000073    1 r INITIALIZING    10.232.7.145   opensearch-master-2
.ds-system-000073    2 r INITIALIZING    10.232.182.135 opensearch-master-1
.ds-audit-000001     2 r INITIALIZING    10.232.7.145   opensearch-master-2

You can additionally analyze the exact level of throttling and the current CPU usage on the Kubernetes Containers dashboard in Grafana.

Workaround:

Verify the currently configured CPU requests and limits for the opensearch containers:

kubectl -n stacklight get statefulset.apps/opensearch-master -o jsonpath="{.spec.template.spec.containers[?(@.name=='opensearch')].resources}"

Example of system response:

{"limits":{"cpu":"600m","memory":"8Gi"},"requests":{"cpu":"500m","memory":"6Gi"}}

In the example above, the CPU request is 500m and the CPU limit is 600m.

Increase the CPU limit to a reasonably high number.

For example, the default CPU limit for the clusters with the clusterSize:large parameter set was increased from 8000m to 12000m for StackLight in Container Cloud 2.27.0 (Cluster releases 17.2.0 and 16.2.0).

Note

For details, on the clusterSize parameter, see MOSK Operations Guide: StackLight configuration parameters - Cluster size.

If the defaults are already overridden on the affected cluster using the resourcesPerClusterSize or resources parameters as described in MOSK Operations Guide: StackLight configuration parameters - Resource limits, then the exact recommended number depends on the currently set limit.

Mirantis recommends increasing the limit by 50%. If it does not resolve the issue, another increase iteration will be required.
When you select the required CPU limit, increase it as described in MOSK Operations Guide: StackLight configuration parameters - Resource limits.

If the CPU limit for the opensearch component is already set, increase it in the Cluster object for the opensearch parameter. Otherwise, the default StackLight limit is used. In this case, increase the CPU limit for the opensearch component using the resources parameter.
Wait until all opensearch-master pods are recreated with the new CPU limits and become running and ready.

To verify the current CPU limit for every opensearch container in every opensearch-master pod separately:
```
kubectl -n stacklight get pod/opensearch-master-<podSuffixNumber> -o jsonpath="{.spec.containers[?(@.name=='opensearch')].resources}"
```
In the command above, replace <podSuffixNumber> with the name of the pod suffix. For example, pod/opensearch-master-0 or pod/opensearch-master-2.

Example of system response:
```
{"limits":{"cpu":"900m","memory":"8Gi"},"requests":{"cpu":"500m","memory":"6Gi"}}
```
The waiting time may take up to 20 minutes depending on the cluster size.

If the KubeContainersCPUThrottlingHigh alert is still firing, proceed with another iteration of the CPU limit increase.

[40020] Rollover policy update is not appllied to the current index¶

Fixed in 17.2.0, 16.2.0, 17.1.6, 16.1.6

While updating rollover_policy for the current system* and audit* data streams, the update is not applied to indices.

2024-05-31 13:16:04,459 ERROR   Failed to complete action: delete_indices.  <class 'curator.exceptions.FailedExecution'>: Exception encountered.  Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: RequestError(400, 'illegal_argument_exception', 'index [.ds-audit-000001] is the write index for data stream [audit] and cannot be deleted')

2024-05-31 13:16:04,459 ERROR   Failed to complete action: delete_indices.  <class 'curator.exceptions.FailedExecution'>: Exception encountered.  Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: RequestError(400, 'illegal_argument_exception', 'index [.ds-system-000001] is the write index for data stream [system] and cannot be deleted')

Note

Instead of .ds-audit-000001 or .ds-system-000001 index names, similar names can be present with the same prefix but different suffix numbers.

If the above mentioned alert and errors are present, an immediate action is required, because it indicates that the corresponding index size has already exceeded the space allocated for the index.

To verify that the cluster is affected:

Caution

Verify and apply the workaround to both index patterns, system and audit, separately.

If one of indices is affected, the second one is most likely affected as well. Although in rare cases, only one index may be affected.

kubectl exec -it pod/opensearch-master-0 -n stacklight -c opensearch -- bash

Verify that the rollover policy is present:
- system:
```
curl localhost:9200/_plugins/_ism/policies/system_rollover_policy
```
- audit:
```
curl localhost:9200/_plugins/_ism/policies/audit_rollover_policy
```
The cluster is affected if the rollover policy is missing. Otherwise, proceed to the following step.

Verify the system response from the previous step. For example:

{"_id":"system_rollover_policy","_version":7229,"_seq_no":42362,"_primary_term":28,"policy":{"policy_id":"system_rollover_policy","description":"system index rollover policy.","last_updated_time":1708505222430,"schema_version":19,"error_notification":null,"default_state":"rollover","states":[{"name":"rollover","actions":[{"retry":{"count":3,"backoff":"exponential","delay":"1m"},"rollover":{"min_size":"14746mb","copy_alias":false}}],"transitions":[]}],"ism_template":[{"index_patterns":["system*"],"priority":200,"last_updated_time":1708505222430}]}}

Verify and capture the following items separately for every policy:

The _seq_no and _primary_term values
The rollover policy threshold, which is defined in policy.states[0].actions[0].rollover.min_size

List indices:

system:

curl localhost:9200/_cat/indices | grep system

Example of system response:

[...]
green open .ds-system-000001   FjglnZlcTKKfKNbosaE9Aw 2 1 1998295  0   1gb 507.9mb

audit:

curl localhost:9200/_cat/indices | grep audit

Example of system response:

[...]
green open .ds-audit-000001   FjglnZlcTKKfKNbosaE9Aw 2 1 1998295  0   1gb 507.9mb

Select the index with the highest number and verify the rollover policy attached to the index:
- system:
```
curl localhost:9200/_plugins/_ism/explain/.ds-system-000001
```
- audit:
```
curl localhost:9200/_plugins/_ism/explain/.ds-audit-000001
```
- If the rollover policy is not attached, the cluster is affected.
- If the rollover policy is attached but _seq_no and _primary_term numbers do not match the previously captured ones, the cluster is affected.
- If the index size drastically exceeds the defined threshold of the rollover policy (which is the previously captured min_size), the cluster is most probably affected.

Workaround:

kubectl exec -it pod/opensearch-master-0 -n stacklight -c opensearch -- bash

If the policy is attached to the index but has different _seq_no and _primary_term, remove the policy from the index:

Note

Use the index with the highest number in the name, which was captured during verification procedure.
- system:
```
curl -XPOST localhost:9200/_plugins/_ism/remove/.ds-system-000001
```
- audit:
```
curl -XPOST localhost:9200/_plugins/_ism/remove/.ds-audit-000001
```

Re-add the policy:

system:

curl -XPOST -H "Content-type: application/json" localhost:9200/_plugins/_ism/add/system* -d'{"policy_id":"system_rollover_policy"}'

audit:

curl -XPOST -H "Content-type: application/json" localhost:9200/_plugins/_ism/add/audit* -d'{"policy_id":"audit_rollover_policy"}'

Perform again the last step of the cluster verification procedure provided above and make sure that the policy is attached to the index and has the same _seq_no and _primary_term.

If the index size drastically exceeds the defined threshold of the rollover policy (which is the previously captured min_size), wait up to 15 minutes and verify that the additional index is created with the consecutive number in the index name. For example:
- system: if you applied changes to .ds-system-000001, wait until .ds-system-000002 is created.
- audit: if you applied changes to .ds-audit-000001, wait until .ds-audit-000002 is created.
If such index is not created, escalate the issue to Mirantis support.

Container Cloud web UI¶

[41806] Configuration of a management cluster fails without Keycloak settings¶

Fixed in 17.1.4 and 16.1.4

During configuration of a management cluster settings using the Configure cluster web UI menu, updating the Keycloak Truststore settings is mandatory, despite being optional.

As a workaround, update the management cluster using the API or CLI.

Artifacts¶

This section lists the artifacts of components included in the Container Cloud patch release 2.26.2. For artifacts of the Cluster releases introduced in 2.26.2, see patch Cluster releases 17.1.2 and 16.1.2.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries ^Updated	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20240324195604
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20240324195604
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.39.19.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.39.19.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.39.19.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.39.19.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.39.19.tgz
	local-volume-provisioner	https://binary.mirantis.com/core/helm/local-volume-provisioner-1.39.19.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.39.19.tgz
Docker images	ambasador ^Updated	mirantis.azurecr.io/core/external/nginx:1.39.19
	baremetal-dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-26-alpine-20240325100252
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-2-26-alpine-20240325093002
	bm-collective	mirantis.azurecr.io/bm/bm-collective:base-2-26-alpine-20240129155244
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.39.19
	ironic	mirantis.azurecr.io/openstack/ironic:yoga-jammy-20240226060024
	ironic-inspector	mirantis.azurecr.io/openstack/ironic-inspector:yoga-jammy-20240226060024
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20240117102150
	kaas-ipam	mirantis.azurecr.io/bm/kaas-ipam:base-2-26-alpine-20240129213142
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-55b02f7-20231019172556
	mariadb ^Updated	mirantis.azurecr.io/general/mariadb:10.6.14-focal-20240311120505
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.24.0-47-gf77368e
	metallb-controller	mirantis.azurecr.io/bm/metallb/controller:v0.13.12-31212f9e-amd64
	metallb-speaker	mirantis.azurecr.io/bm/metallb/speaker:v0.13.12-31212f9e-amd64
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20240129163811

Core artifacts¶

Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.39.19.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.39.19.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.39.19.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.39.19.tgz
	byo-credentials-controller	https://binary.mirantis.com/core/helm/byo-credentials-controller-1.39.19.tgz
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.39.19.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.39.19.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.39.19.tgz
	cinder-csi-plugin	https://binary.mirantis.com/core/helm/cinder-csi-plugin-1.39.19.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.39.19.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.39.19.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.39.19.tgz
	host-os-modules-controller	https://binary.mirantis.com/core/helm/host-os-modules-controller-1.39.19.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.39.19.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.39.19.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.39.19.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.39.19.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.39.19.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.39.19.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.39.19.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.39.19.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.39.19.tgz
	metrics-server	https://binary.mirantis.com/core/helm/metrics-server-1.39.19.tgz
	openstack-cloud-controller-manager	https://binary.mirantis.com/core/helm/openstack-cloud-controller-manager-1.39.19.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.39.19.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.39.19.tgz
	policy-controller	https://binary.mirantis.com/core/helm/policy-controller-1.39.19.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.39.19.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.39.19.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.39.19.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.39.19.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.39.19.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.39.19.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.39.19.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.39.19.tgz
	vsphere-cloud-controller-manager	https://binary.mirantis.com/core/helm/vsphere-cloud-controller-manager-1.39.19.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.39.19.tgz
	vsphere-csi-plugin	https://binary.mirantis.com/core/helm/vsphere-csi-plugin-1.39.19.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.39.19.tgz
	vsphere-vm-template-controller	https://binary.mirantis.com/core/helm/vsphere-vm-template-controller-1.39.19.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.39.19
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.39.19
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.39.19
	byo-credentials-controller ^Updated	mirantis.azurecr.io/core/byo-credentials-controller:1.39.19
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.39.19
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-5
	cinder-csi-plugin	mirantis.azurecr.io/lcm/kubernetes/cinder-csi-plugin:v1.27.2-13
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.39.19
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.39.19
	csi-attacher	mirantis.azurecr.io/lcm/k8scsi/csi-attacher:v4.2.0-4
	csi-node-driver-registrar	mirantis.azurecr.io/lcm/k8scsi/csi-node-driver-registrar:v2.7.0-4
	csi-provisioner	mirantis.azurecr.io/lcm/k8scsi/csi-provisioner:v3.4.1-4
	csi-resizer	mirantis.azurecr.io/lcm/k8scsi/csi-resizer:v1.7.0-4
	csi-snapshotter	mirantis.azurecr.io/lcm/k8scsi/csi-snapshotter:v6.2.1-mcc-3
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.39.19
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.39.19
	host-os-modules-controller ^Updated	mirantis.azurecr.io/core/host-os-modules-controller:1.39.19
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.39.19
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.39.19
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.39.19
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.39.19
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.39.19
	livenessprobe	mirantis.azurecr.io/lcm/k8scsi/livenessprobe:v2.9.0-4
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.39.19
	mcc-haproxy	mirantis.azurecr.io/lcm/mcc-haproxy:v0.24.0-47-gf77368e
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.24.0-47-gf77368e
	metrics-server	mirantis.azurecr.io/lcm/metrics-server-amd64:v0.6.3-6
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.39.19
	openstack-cloud-controller-manager ^Updated	mirantis.azurecr.io/lcm/kubernetes/openstack-cloud-controller-manager:v1.27.2-13
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.39.19
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.39.19
	policy-controller ^Updated	mirantis.azurecr.io/core/policy-controller:1.39.19
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.39.19
	proxy-controller ^Updated	mirantis.azurecr.io/core/proxy-controller:1.39.19
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.39.19
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-9
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.39.19
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.39.19
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.39.19
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-10-g24a0d69
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.39.19
	vsphere-cloud-controller-manager	mirantis.azurecr.io/lcm/kubernetes/vsphere-cloud-controller-manager:v1.27.0-5
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.39.19
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.39.19
	vsphere-csi-driver	mirantis.azurecr.io/lcm/kubernetes/vsphere-csi-driver:v3.0.2-1
	vsphere-csi-syncer	mirantis.azurecr.io/lcm/kubernetes/vsphere-csi-syncer:v3.0.2-1
	vsphere-vm-template-controller ^Updated	mirantis.azurecr.io/core/vsphere-vm-template-controller:1.39.19

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts ^Updated	iam	https://binary.mirantis.com/core/helm/iam-1.39.19.tgz
Docker images	kubectl ^Updated	mirantis.azurecr.io/stacklight/kubectl:1.22-20240221023016
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-55b02f7-20231019172556
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.14-focal-20231127070342
	mcc-keycloak ^Updated	mirantis.azurecr.io/iam/mcc-keycloak:23.0.6-20240216125244

See also

Patch releases

2.26.1¶

The Container Cloud patch release 2.26.1, which is based on the 2.26.0 major release, provides the following updates:

Support for the patch Cluster releases 16.1.1 and 17.1.1 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 24.1.1.
Delivery mechanism for CVE fixes on Ubuntu in bare metal clusters that includes update of Ubuntu kernel minor version. For details, see Enhancements.
Security fixes for CVEs in images.

For main deliverables of the parent Container Cloud release of 2.26.1, refer to 2.26.0.

Enhancements¶

This section outlines new features and enhancements introduced in the Container Cloud patch release 2.26.1 along with Cluster releases 17.1.1 and 16.1.1.

Delivery mechanism for CVE fixes on Ubuntu in bare metal clusters¶

Introduced the ability to update Ubuntu packages including kernel minor version update, when available in a Cluster release, for both management and managed bare metal clusters to address CVE issues on a host operating system.

On management clusters, the update of Ubuntu mirror along with the update of minor kernel version occurs automatically with cordon-drain and reboot of machines.
On managed clusters, the update of Ubuntu mirror along with the update of minor kernel version applies during a manual cluster update without automatic cordon-drain and reboot of machines. After a managed cluster update, all cluster machines have the reboot is required notification. You can manually handle the reboot of machines during a convenient maintenance window using GracefulRebootRequest.

Learn more

Artifacts¶

This section lists the artifacts of components included in the Container Cloud patch release 2.26.1. For artifacts of the Cluster releases introduced in 2.26.1, see patch Cluster releases 17.1.1 and 16.1.1.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Bare metal artifacts¶
Artifact	Component	Path
Binaries ^Updated	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20240302181430
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20240302181430
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-155-1882779.tgz
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.39.15.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.39.15.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.39.15.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.39.15.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.39.15.tgz
	local-volume-provisioner	https://binary.mirantis.com/core/helm/local-volume-provisioner-1.39.15.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.39.15.tgz
Docker images	ambasador ^Updated	mirantis.azurecr.io/core/external/nginx:1.39.15
	baremetal-dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-26-alpine-20240226130438
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-2-26-alpine-20240226130310
	bm-collective	mirantis.azurecr.io/bm/bm-collective:base-2-26-alpine-20240129155244
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.39.15
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:yoga-jammy-20240226060024
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:yoga-jammy-20240226060024
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20240117102150
	kaas-ipam	mirantis.azurecr.io/bm/kaas-ipam:base-2-26-alpine-20240129213142
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-55b02f7-20231019172556
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.14-focal-20231127070342
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.24.0-47-gf77368e
	metallb-controller	mirantis.azurecr.io/bm/metallb/controller:v0.13.12-31212f9e-amd64
	metallb-speaker	mirantis.azurecr.io/bm/metallb/speaker:v0.13.12-31212f9e-amd64
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20240129163811

Core artifacts¶

Core artifacts¶
Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.39.15.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.39.15.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.39.15.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.39.15.tgz
	byo-credentials-controller	https://binary.mirantis.com/core/helm/byo-credentials-controller-1.39.15.tgz
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.39.15.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.39.15.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.39.15.tgz
	cinder-csi-plugin	https://binary.mirantis.com/core/helm/cinder-csi-plugin-1.39.15.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.39.15.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.39.15.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.39.15.tgz
	host-os-modules-controller	https://binary.mirantis.com/core/helm/host-os-modules-controller-1.39.15.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.39.15.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.39.15.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.39.15.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.39.15.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.39.15.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.39.15.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.39.15.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.39.15.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.39.15.tgz
	metrics-server	https://binary.mirantis.com/core/helm/metrics-server-1.39.15.tgz
	openstack-cloud-controller-manager	https://binary.mirantis.com/core/helm/openstack-cloud-controller-manager-1.39.15.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.39.15.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.39.15.tgz
	policy-controller	https://binary.mirantis.com/core/helm/policy-controller-1.39.15.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.39.15.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.39.15.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.39.15.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.39.15.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.39.15.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.39.15.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.39.15.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.39.15.tgz
	vsphere-cloud-controller-manager	https://binary.mirantis.com/core/helm/vsphere-cloud-controller-manager-1.39.15.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.39.15.tgz
	vsphere-csi-plugin	https://binary.mirantis.com/core/helm/vsphere-csi-plugin-1.39.15.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.39.15.tgz
	vsphere-vm-template-controller	https://binary.mirantis.com/core/helm/vsphere-vm-template-controller-1.39.15.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.39.15
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.39.15
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.39.15
	byo-credentials-controller ^Updated	mirantis.azurecr.io/core/byo-credentials-controller:1.39.15
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.39.15
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-5
	cinder-csi-plugin ^Updated	mirantis.azurecr.io/lcm/kubernetes/cinder-csi-plugin:v1.27.2-13
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.39.15
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.39.15
	csi-attacher	mirantis.azurecr.io/lcm/k8scsi/csi-attacher:v4.2.0-4
	csi-node-driver-registrar	mirantis.azurecr.io/lcm/k8scsi/csi-node-driver-registrar:v2.7.0-4
	csi-provisioner	mirantis.azurecr.io/lcm/k8scsi/csi-provisioner:v3.4.1-4
	csi-resizer	mirantis.azurecr.io/lcm/k8scsi/csi-resizer:v1.7.0-4
	csi-snapshotter	mirantis.azurecr.io/lcm/k8scsi/csi-snapshotter:v6.2.1-mcc-3
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.39.15
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.39.15
	host-os-modules-controller ^Updated	mirantis.azurecr.io/core/host-os-modules-controller:1.39.15
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.39.15
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.39.15
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.39.15
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.39.15
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.39.15
	livenessprobe	mirantis.azurecr.io/lcm/k8scsi/livenessprobe:v2.9.0-4
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.39.15
	mcc-haproxy ^Updated	mirantis.azurecr.io/lcm/mcc-haproxy:v0.24.0-47-gf77368e
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.24.0-47-gf77368e
	metrics-server	mirantis.azurecr.io/lcm/metrics-server-amd64:v0.6.3-6
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.39.15
	openstack-cloud-controller-manager ^Updated	mirantis.azurecr.io/lcm/kubernetes/openstack-cloud-controller-manager:v1.27.2-13
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.39.15
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.39.15
	policy-controller ^Updated	mirantis.azurecr.io/core/policy-controller:1.39.15
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.39.15
	proxy-controller ^Updated	mirantis.azurecr.io/core/proxy-controller:1.39.15
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.39.15
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-9
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.39.15
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.39.15
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.39.15
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-10-g24a0d69
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.39.15
	vsphere-cloud-controller-manager	mirantis.azurecr.io/lcm/kubernetes/vsphere-cloud-controller-manager:v1.27.0-5
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.39.15
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.39.15
	vsphere-csi-driver	mirantis.azurecr.io/lcm/kubernetes/vsphere-csi-driver:v3.0.2-1
	vsphere-csi-syncer	mirantis.azurecr.io/lcm/kubernetes/vsphere-csi-syncer:v3.0.2-1
	vsphere-vm-template-controller ^Updated	mirantis.azurecr.io/core/vsphere-vm-template-controller:1.39.15

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts	iam	https://binary.mirantis.com/core/helm/iam-1.39.15.tgz
Docker images	kubectl	mirantis.azurecr.io/stacklight/kubectl:1.22-20240105023016
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-55b02f7-20231019172556
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.14-focal-20231127070342
	mcc-keycloak	mirantis.azurecr.io/iam/mcc-keycloak:23.0.3-1

Security notes¶

The table below includes the total numbers of addressed unique and common CVEs in images by product component since the Container Cloud 2.26.0 major release. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	High	Total
Ceph	Unique	1	1
Ceph	Common	3	3
Kaas core	Unique	6	6
Kaas core	Common	27	27
StackLight	Unique	15	15
StackLight	Common	51	51

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 24.1.1: Security notes.

Addressed issues¶

The following issues have been addressed in the Container Cloud patch release 2.26.1 along with the patch Cluster releases 17.1.1 and 16.1.1.

[39330] [StackLight] Fixed the issue with the OpenSearch cluster being stuck due to initializing replica shards.
[39220] [StackLight] Fixed the issue with Patroni failure due to no limit configuration for the max_timelines_history parameter.
[39080] [StackLight] Fixed the issue with the OpenSearchClusterStatusWarning alert firing during cluster upgrade if StackLight is deployed in the HA mode.
[38970] [StackLight] Fixed the issue with the Logs dashboard in the OpenSearch Dashboards web UI not working for the system index.
[38937] [StackLight] Fixed the issue with the View logs in OpenSearch Dashboards link not working in the Grafana web UI.
[40747] [vSphere] Fixed the issue with the unsupported Cluster release being available for greenfield vSphere-based managed cluster deployments in the drop-down menu of the cluster creation window in the Container Cloud web UI.
[40036] [LCM] Fixed the issue causing nodes to remain in the Kubernetes cluster when the corresponding Machine object is disabled during cluster update.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.26.1 including the Cluster releases 17.1.1 and 16.1.1.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[46245] Lack of access permissions for HOC and HOCM objects¶

Fixed in 2.28.0 (17.3.0 and 16.3.0)

kubectl --kubeconfig ~/.kube/mgmt-config get hocm

Error from server (Forbidden): hostosconfigurationmodules.kaas.mirantis.com is forbidden:
User "2d74348b-5669-4c65-af31-6c05dbedac5f" cannot list resource "hostosconfigurationmodules"
in API group "kaas.mirantis.com" at the cluster scope: access denied

Workaround:

Modify the global-admin role by adding a new entry with the following contents to the rules list:

kubectl edit clusterroles kaas-global-admin

- apiGroups: [kaas.mirantis.com]
  resources: [hostosconfigurationmodules]
  verbs: ['*']

For each Container Cloud project, modify the kaas-operator role by adding a new entry with the following contents to the rules list:

kubectl -n <projectName> edit roles kaas-operator

- apiGroups: [kaas.mirantis.com]
  resources: [hostosconfigurations]
  verbs: ['*']

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[41305] DHCP responses are lost between dnsmasq and dhcp-relay pods¶

Fixed in 2.28.0 (17.3.0 and 16.3.0)

The issue was observed on environments having management cluster nodes configured with a single L2 segment used for all network traffic (PXE and LCM/management networks).

To verify whether the cluster is affected:

Verify whether the dnsmasq and dhcp-relay pods run on the same node in the management cluster:

kubectl -n kaas get pods -o wide| grep -e "dhcp\|dnsmasq"

Example of system response:

dhcp-relay-7d85f75f76-5vdw2   2/2   Running   2 (36h ago)   36h   10.10.0.122     kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>
dnsmasq-8f4b484b4-slhbd       5/5   Running   1 (36h ago)   36h   10.233.123.75   kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>

If this is the case, proceed to the workaround below.

Workaround:

Make sure that at least two management cluster nodes are schedulable:

kubectl get node

Example of a positive system response:

NAME                                             STATUS   ROLES    AGE   VERSION
kaas-node-bcedb87b-b3ce-46a4-a4ca-ea3068689e40   Ready    master   37h   v1.27.10-mirantis-1
kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   Ready    master   37h   v1.27.10-mirantis-1
kaas-node-ad5a6f51-b98f-43c3-91d5-55fed3d0ff21   Ready    master   37h   v1.27.10-mirantis-1

Delete the dhcp-relay pod:

kubectl -n kaas delete pod <dhcp-relay-xxxxx>

Verify that the dnsmasq and dhcp-relay pods are scheduled into different nodes:

kubectl -n kaas get pods -o wide| grep -e "dhcp\|dnsmasq"

Example of a positive system response:

dhcp-relay-7d85f75f76-rkv03   2/2   Running   0             49s   10.10.0.121     kaas-node-bcedb87b-b3ce-46a4-a4ca-ea3068689e40   <none>   <none>
dnsmasq-8f4b484b4-slhbd       5/5   Running   1 (37h ago)   37h   10.233.123.75   kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

LCM¶

[41540] LCM Agent cannot grab storage information on a host¶

Fixed in 17.1.5 and 16.1.5

{"level":"error","ts":"2024-05-02T12:26:10Z","logger":"agent", \
"msg":"get hardware details", \
"host":"kaas-node-548b2861-aed0-41c9-8ff2-10c5476b000b", \
"error":"new storage info: get disk info \"nvme0c0n1\": \
invoke command: exit status 1","errorVerbose":"exit status 1

As a workaround, on the affected node, create a symlink for any device indicated in lcm-agent logs. For example:

ln -sfn /dev/nvme0n1 /dev/nvme0c0n1

[40811] Pod is stuck in the Terminating state on the deleted node¶

Fixed in 17.1.3 and 16.1.3

During deletion of a machine, the related DaemonSet Pod can remain on the deleted node in the Terminating state. As a workaround, manually delete the Pod:

kubectl delete pod -n <podNamespace> <podName>

[39437] Failure to replace a master node on a Container Cloud cluster¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

During the replacement of a master node on a cluster of any type, the process may get stuck with Kubelet's NodeReady condition is Unknown in the machine status on the remaining master nodes.

As a workaround, log in on the affected node and run the following command:

docker restart ucp-kubelet

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

[30294] Replacement of a master node is stuck on the calico-node Pod start¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a master node on a cluster of any type, the calico-node Pod fails to start on a new node that has the same IP address as the node being replaced.

Workaround:

From a CLI with an MKE client bundle, create a shell alias to start calicoctl using the mirantis/ucp-dsinfo image:

Since MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/key.pem \
-e ETCD_CA_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/ca.pem \
-e ETCD_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/cert.pem \
-v /var/run/calico:/var/run/calico \
-v /var/lib/docker/volumes/ucp-kv-certs/_data:/var/lib/docker/volumes/ucp-kv-certs/_data:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl \
"

Before MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
-e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
-e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
-v /var/run/calico:/var/run/calico \
-v ucp-node-certs:/ucp-node-certs:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl --allow-version-mismatch \
"

In the above command, replace the following values with the corresponding settings of the affected cluster:

<etcdEndpoint> is the etcd endpoint defined in the Calico configuration file. For example, ETCD_ENDPOINTS=127.0.0.1:12378
<mkeVersion> is the MKE version installed on your cluster. For example, mirantis/ucp-dsinfo:3.5.7.

Verify the node list on the cluster:
```
kubectl get node
```
Compare this list with the node list in Calico to identify the old node:
```
calicoctl get node -o wide
```

Remove the old node from Calico:

calicoctl delete node kaas-node-<nodeID>

[5782] Manager machine fails to be deployed during node replacement¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a manager machine, the following problems may occur:

The system adds the node to Docker swarm but not to Kubernetes
The node Deployment gets stuck with failed RethinkDB health checks

Workaround:

Delete the failed node.
Wait for the MKE cluster to become healthy. To monitor the cluster status:
1. Log in to the MKE web UI as described in Connect to the Mirantis Kubernetes Engine web UI.
2. Monitor the cluster status as described in MKE Operations Guide: Monitor an MKE cluster with the MKE web UI.
Deploy a new node.

[5568] The calico-kube-controllers Pod fails to clean up resources¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During the unsafe or forced deletion of a manager machine running the calico-kube-controllers Pod in the kube-system namespace, the following issues occur:

The calico-kube-controllers Pod fails to clean up resources associated with the deleted node
The calico-node Pod may fail to start up on a newly created node if the machine is provisioned with the same IP address as the deleted machine had

As a workaround, before deletion of the node running the calico-kube-controllers Pod, cordon and drain the node:

kubectl cordon <nodeName>
kubectl drain <nodeName>

Ceph¶

[41819] Graceful cluster reboot is blocked by the Ceph ClusterWorkloadLocks¶

Fixed in 2.27.0 (17.2.0 and 16.2.0)

During graceful reboot of a cluster with Ceph enabled, the reboot is blocked with the following message in the MiraCephMaintenance object status:

message: ClusterMaintenanceRequest found, Ceph Cluster is not ready to upgrade,
 delaying cluster maintenance

As a workaround, add the following snippet to the cephFS section under metadataServer in the spec section of <kcc-name>.yaml in the Ceph cluster:

cephClusterSpec:
  sharedFilesystem:
    cephFS:
    - name: cephfs-store
      metadataServer:
        activeCount: 1
        healthCheck:
          livenessProbe:
            probe:
              failureThreshold: 5
              initialDelaySeconds: 30
              periodSeconds: 30
              successThreshold: 1
              timeoutSeconds: 5

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

StackLight¶

[42304] Failure of shard relocation in the OpenSearch cluster¶

Fixed in 17.2.0, 16.2.0, 17.1.6, 16.1.6

{created_by_kind="StatefulSet",created_by_name="opensearch-master",namespace="stacklight"}

Caution

To verify that the initialization of shards is ongoing:

kubectl exec -it pod/opensearch-master-0 -n stacklight -c opensearch -- bash

curl "http://localhost:9200/_cat/shards" | grep INITIALIZING

Example of system response:

.ds-system-000072    2 r INITIALIZING    10.232.182.135 opensearch-master-1
.ds-system-000073    1 r INITIALIZING    10.232.7.145   opensearch-master-2
.ds-system-000073    2 r INITIALIZING    10.232.182.135 opensearch-master-1
.ds-audit-000001     2 r INITIALIZING    10.232.7.145   opensearch-master-2

You can additionally analyze the exact level of throttling and the current CPU usage on the Kubernetes Containers dashboard in Grafana.

Workaround:

Verify the currently configured CPU requests and limits for the opensearch containers:

kubectl -n stacklight get statefulset.apps/opensearch-master -o jsonpath="{.spec.template.spec.containers[?(@.name=='opensearch')].resources}"

Example of system response:

{"limits":{"cpu":"600m","memory":"8Gi"},"requests":{"cpu":"500m","memory":"6Gi"}}

In the example above, the CPU request is 500m and the CPU limit is 600m.

Increase the CPU limit to a reasonably high number.

For example, the default CPU limit for the clusters with the clusterSize:large parameter set was increased from 8000m to 12000m for StackLight in Container Cloud 2.27.0 (Cluster releases 17.2.0 and 16.2.0).

Note

For details, on the clusterSize parameter, see MOSK Operations Guide: StackLight configuration parameters - Cluster size.

If the defaults are already overridden on the affected cluster using the resourcesPerClusterSize or resources parameters as described in MOSK Operations Guide: StackLight configuration parameters - Resource limits, then the exact recommended number depends on the currently set limit.

Mirantis recommends increasing the limit by 50%. If it does not resolve the issue, another increase iteration will be required.
When you select the required CPU limit, increase it as described in MOSK Operations Guide: StackLight configuration parameters - Resource limits.

If the CPU limit for the opensearch component is already set, increase it in the Cluster object for the opensearch parameter. Otherwise, the default StackLight limit is used. In this case, increase the CPU limit for the opensearch component using the resources parameter.
Wait until all opensearch-master pods are recreated with the new CPU limits and become running and ready.

To verify the current CPU limit for every opensearch container in every opensearch-master pod separately:
```
kubectl -n stacklight get pod/opensearch-master-<podSuffixNumber> -o jsonpath="{.spec.containers[?(@.name=='opensearch')].resources}"
```
In the command above, replace <podSuffixNumber> with the name of the pod suffix. For example, pod/opensearch-master-0 or pod/opensearch-master-2.

Example of system response:
```
{"limits":{"cpu":"900m","memory":"8Gi"},"requests":{"cpu":"500m","memory":"6Gi"}}
```
The waiting time may take up to 20 minutes depending on the cluster size.

If the KubeContainersCPUThrottlingHigh alert is still firing, proceed with another iteration of the CPU limit increase.

[40020] Rollover policy update is not appllied to the current index¶

Fixed in 17.2.0, 16.2.0, 17.1.6, 16.1.6

While updating rollover_policy for the current system* and audit* data streams, the update is not applied to indices.

2024-05-31 13:16:04,459 ERROR   Failed to complete action: delete_indices.  <class 'curator.exceptions.FailedExecution'>: Exception encountered.  Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: RequestError(400, 'illegal_argument_exception', 'index [.ds-audit-000001] is the write index for data stream [audit] and cannot be deleted')

2024-05-31 13:16:04,459 ERROR   Failed to complete action: delete_indices.  <class 'curator.exceptions.FailedExecution'>: Exception encountered.  Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: RequestError(400, 'illegal_argument_exception', 'index [.ds-system-000001] is the write index for data stream [system] and cannot be deleted')

Note

Instead of .ds-audit-000001 or .ds-system-000001 index names, similar names can be present with the same prefix but different suffix numbers.

If the above mentioned alert and errors are present, an immediate action is required, because it indicates that the corresponding index size has already exceeded the space allocated for the index.

To verify that the cluster is affected:

Caution

Verify and apply the workaround to both index patterns, system and audit, separately.

If one of indices is affected, the second one is most likely affected as well. Although in rare cases, only one index may be affected.

kubectl exec -it pod/opensearch-master-0 -n stacklight -c opensearch -- bash

Verify that the rollover policy is present:
- system:
```
curl localhost:9200/_plugins/_ism/policies/system_rollover_policy
```
- audit:
```
curl localhost:9200/_plugins/_ism/policies/audit_rollover_policy
```
The cluster is affected if the rollover policy is missing. Otherwise, proceed to the following step.

Verify the system response from the previous step. For example:

{"_id":"system_rollover_policy","_version":7229,"_seq_no":42362,"_primary_term":28,"policy":{"policy_id":"system_rollover_policy","description":"system index rollover policy.","last_updated_time":1708505222430,"schema_version":19,"error_notification":null,"default_state":"rollover","states":[{"name":"rollover","actions":[{"retry":{"count":3,"backoff":"exponential","delay":"1m"},"rollover":{"min_size":"14746mb","copy_alias":false}}],"transitions":[]}],"ism_template":[{"index_patterns":["system*"],"priority":200,"last_updated_time":1708505222430}]}}

Verify and capture the following items separately for every policy:

The _seq_no and _primary_term values
The rollover policy threshold, which is defined in policy.states[0].actions[0].rollover.min_size

List indices:

system:

curl localhost:9200/_cat/indices | grep system

Example of system response:

[...]
green open .ds-system-000001   FjglnZlcTKKfKNbosaE9Aw 2 1 1998295  0   1gb 507.9mb

audit:

curl localhost:9200/_cat/indices | grep audit

Example of system response:

[...]
green open .ds-audit-000001   FjglnZlcTKKfKNbosaE9Aw 2 1 1998295  0   1gb 507.9mb

Select the index with the highest number and verify the rollover policy attached to the index:
- system:
```
curl localhost:9200/_plugins/_ism/explain/.ds-system-000001
```
- audit:
```
curl localhost:9200/_plugins/_ism/explain/.ds-audit-000001
```
- If the rollover policy is not attached, the cluster is affected.
- If the rollover policy is attached but _seq_no and _primary_term numbers do not match the previously captured ones, the cluster is affected.
- If the index size drastically exceeds the defined threshold of the rollover policy (which is the previously captured min_size), the cluster is most probably affected.

Workaround:

kubectl exec -it pod/opensearch-master-0 -n stacklight -c opensearch -- bash

If the policy is attached to the index but has different _seq_no and _primary_term, remove the policy from the index:

Note

Use the index with the highest number in the name, which was captured during verification procedure.
- system:
```
curl -XPOST localhost:9200/_plugins/_ism/remove/.ds-system-000001
```
- audit:
```
curl -XPOST localhost:9200/_plugins/_ism/remove/.ds-audit-000001
```

Re-add the policy:

system:

curl -XPOST -H "Content-type: application/json" localhost:9200/_plugins/_ism/add/system* -d'{"policy_id":"system_rollover_policy"}'

audit:

curl -XPOST -H "Content-type: application/json" localhost:9200/_plugins/_ism/add/audit* -d'{"policy_id":"audit_rollover_policy"}'

Perform again the last step of the cluster verification procedure provided above and make sure that the policy is attached to the index and has the same _seq_no and _primary_term.

If the index size drastically exceeds the defined threshold of the rollover policy (which is the previously captured min_size), wait up to 15 minutes and verify that the additional index is created with the consecutive number in the index name. For example:
- system: if you applied changes to .ds-system-000001, wait until .ds-system-000002 is created.
- audit: if you applied changes to .ds-audit-000001, wait until .ds-audit-000002 is created.
If such index is not created, escalate the issue to Mirantis support.

Container Cloud web UI¶

[41806] Configuration of a management cluster fails without Keycloak settings¶

Fixed in 17.1.4 and 16.1.4

During configuration of a management cluster settings using the Configure cluster web UI menu, updating the Keycloak Truststore settings is mandatory, despite being optional.

As a workaround, update the management cluster using the API or CLI.

See also

Patch releases

2.26.0¶

The Mirantis Container Cloud major release 2.26.0:

Introduces support for the Cluster release 17.1.0 that is based on the Cluster release 16.1.0 and represents Mirantis OpenStack for Kubernetes (MOSK) 24.1.
Introduces support for the Cluster release 16.1.0 that is based on Mirantis Container Runtime (MCR) 23.0.9 and Mirantis Kubernetes Engine (MKE) 3.7.5 with Kubernetes 1.27.
Does not support greenfield deployments on deprecated Cluster releases of the 17.0.x and 16.0.x series. Use the latest available Cluster releases of the series instead.

Caution

Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.26.0.

Enhancements¶

This section outlines new features and enhancements introduced in the Container Cloud release 2.26.0. For the list of enhancements delivered with the Cluster releases introduced by Container Cloud 2.26.0, see 17.1.0 and 16.1.0.

Pre-update inspection of pinned product artifacts in a ‘Cluster’ object¶

To ensure that Container Cloud clusters remain consistently updated with the latest security fixes and product improvements, the Admission Controller has been enhanced. Now, it actively prevents the utilization of pinned custom artifacts for Container Cloud components. Specifically, it blocks a management or managed cluster release update, or any cluster configuration update, for example, adding public keys or proxy, if a Cluster object contains any custom Container Cloud artifacts with global or image-related values overwritten in the helm-releases section, until these values are removed.

Normally, the Container Cloud clusters do not contain pinned artifacts, which eliminates the need for any pre-update actions in most deployments. However, if the update of your cluster is blocked with the invalid HelmReleases configuration error, refer to Update notes: Pre-update actions for details.

Note

In rare cases, if the image-related or global values should be changed, you can use the ClusterRelease or KaaSRelease objects instead. But make sure to update these values manually after every major and patch update.

Note

The pre-update inspection applies only to images delivered by Container Cloud that are overwritten. Any custom images unrelated to the product components are not verified and do not block cluster update.

Learn more

Update notes: Pre-update actions

Disablement of worker machines on managed clusters¶

TechPreview

Implemented the machine disabling API that allows you to seamlessly remove a worker machine from the LCM control of a managed cluster. This action isolates the affected node without impacting other machines in the cluster, effectively eliminating it from the Kubernetes cluster. This functionality proves invaluable in scenarios where a malfunctioning machine impedes cluster updates.

Learn more

Operations Guide: Disable a machine

Day-2 management API for bare metal clusters¶

TechPreview

Added initial Technology Preview support for the HostOSConfiguration and HostOSConfigurationModules custom resources in the bare metal provider. These resources introduce configuration modules that allow managing the operating system of a bare metal host granularly without rebuilding the node from scratch. Such approach prevents workload evacuation and significantly reduces configuration time.

Configuration modules manage various settings of the operating system using Ansible playbooks, adhering to specific schemas and metadata requirements. For description of module format, schemas, and rules, contact Mirantis support.

Warning

For security reasons and to ensure safe and reliable cluster operability, contact Mirantis support to start using these custom resources.

Caution

As long as the feature is still on the development stage, Mirantis highly recommends deleting all HostOSConfiguration objects, if any, before automatic upgrade of the management cluster to Container Cloud 2.27.0 (Cluster release 16.2.0). After the upgrade, you can recreate the required objects using the updated parameters.

Learn more

Strict filtering for devices on bare metal clusters¶

Implemented the strict byID filtering for targeting system disks using specific device options: byPath, serialNumber, and wwn. These options offer a more reliable alternative to the unpredictable byName naming format.

Mirantis recommends adopting these new device naming options when adding new nodes and redeploying existing ones to ensure a predictable and stable device naming schema.

Learn more

API Reference: BareMetalHostProfile resource

Dynamic IP allocation for faster host provisioning¶

Introduced a mechanism in the Container Cloud dnsmasq server to dynamically allocate IP addresses for baremetal hosts during provisioning. This new mechanism replaces sequential IP allocation that includes the ping check with dynamic IP allocation without the ping check. Such behavior significantly increases the amount of baremetal servers that you can provision in parallel, which allows you to streamline the process of setting up a large managed cluster.

Learn more

Support for Kubernetes auditing and profiling on management clusters¶

Added support for the Kubernetes auditing and profiling enablement and configuration on management clusters. The auditing option is enabled by default. You can configure both options using Cluster object of the management cluster.

Note

For managed clusters, you can also configure Kubernetes auditing along with profiling using the Cluster object of a managed cluster.

Learn more

Operations Guide: Configure Kubernetes auditing and profiling

Cleanup of LVM thin pool volumes during cluster provisioning¶

Implemented automatic cleanup of LVM thin pool volumes during the provisioning stage to prevent issues with logical volume detection before removal, which could cause node cleanup failure during cluster redeployment.

Wiping a device or partition before a bare metal cluster deployment¶

Implemented the capability to erase existing data from hardware devices to be used for a bare metal management or managed cluster deployment. Using the new wipeDevice structure, you can either erase an existing partition or remove all existing partitions from a physical device. For these purposes, use the eraseMetadata or eraseDevice option that configures cleanup behavior during configuration of a custom bare metal host profile.

Note

The wipeDevice option replaces the deprecated wipe option that will be removed in one of the following releases. For backward compatibility, any existing wipe: true option is automatically converted to the following structure:

wipeDevice:
  eraseMetadata:
    enabled: True

Learn more

Policy Controller for validating pod image signatures¶

Technology Preview

Introduced initial Technology Preview support for the Policy Controller that validates signatures of pod images. The Policy Controller verifies that images used by the Container Cloud and Mirantis OpenStack for Kubernetes controllers are signed by a trusted authority. The Policy Controller inspects defined image policies that list Docker registries and authorities for signature validation.

Learn more

Security Guide: Container images signing and verification

Configuring trusted certificates for Keycloak¶

Added support for configuring Keycloak truststore using the Container Cloud web UI to allow for a proper validation of client self-signed certificates. The truststore is used to ensure secured connection to identity brokers, LDAP identity providers, and others.

Learn more

Health monitoring of cluster LCM operations¶

Added the LCM Operation condition to monitor health of all LCM operations on a cluster and its machines that is useful during cluster update. You can monitor the status of LCM operations using the the Container Cloud web UI in the status hover menus of a cluster and machine.

Learn more

Operations Guide: Update a managed cluster

Container Cloud web UI improvements for bare metal¶

Reorganized the Container Cloud web UI to optimize the baremetal-based managed cluster deployment and management:

Moved the L2 Templates and Subnets tabs from the Clusters menu to the separate Networks tab on the left sidebar.
Improved the Create Subnet menu by adding configuration for different subnet types.
Reorganized the Baremetal tab in the left sidebar that now contains Hosts, Hosts Profiles, and Credentials tabs.
Implemented the ability to add bare metal host profiles using the web UI.
Moved description of a baremetal host to Host info located in a baremetal host kebab menu on the Hosts page of the Baremetal tab.
Moved description of baremetal host credentials to Credential info located in a credential kebab menu on the Credentials page of the Baremetal tab.

Learn more

Operations Guide: Create and operate a baremetal-based managed cluster

Documentation enhancements¶

On top of continuous improvements delivered to the existing Container Cloud guides, added the documentation on how to export logs from OpenSearch dashboards to CSV.

Learn more

Operations Guide: StackLight operations - Export logs from OpenSearch dashboards to CSV

Addressed issues¶

The following issues have been addressed in the Mirantis Container Cloud release 2.26.0 along with the Cluster releases 17.1.0 and 16.1.0.

Note

This section provides descriptions of issues addressed since the last Container Cloud patch release 2.25.4.

For details on addressed issues in earlier patch releases since 2.25.0, which are also included into the major release 2.26.0, refer to 2.25.x patch releases.

[32761] [LCM] Fixed the issue with node cleanup failing on MOSK clusters due to the Ansible provisioner hanging in a loop while trying to remove LVM thin pool logical volumes, which occurred due to issues with volume detection before removal during cluster redeployment. The issue resolution comprises implementation of automatic cleanup of LVM thin pool volumes during the provisioning stage.
[36924] [LCM] Fixed the issue with Ansible starting to run on nodes of a managed cluster after the mcc-cache certificate is applied on a management cluster.
[37268] [LCM] Fixed the issue with Container Cloud cluster being blocked by a node stuck in the Prepare or Deploy state with error processing package openssh-server. The issue was caused by customizations in /etc/ssh/sshd_config, such as additional Match statements.
[34820] [Ceph] Fixed the issue with the Ceph rook-operator failing to connect to Ceph RADOS Gateway pods on clusters with the Federal Information Processing Standard mode enabled.
[38340] [StackLight] Fixed the issue with Telegraf Docker Swarm timing out while collecting data by increasing its timeout from 10 to 25 seconds.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.26.0 including the Cluster releases 17.1.0 and 16.1.0.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[46245] Lack of access permissions for HOC and HOCM objects¶

Fixed in 2.28.0 (17.3.0 and 16.3.0)

kubectl --kubeconfig ~/.kube/mgmt-config get hocm

Error from server (Forbidden): hostosconfigurationmodules.kaas.mirantis.com is forbidden:
User "2d74348b-5669-4c65-af31-6c05dbedac5f" cannot list resource "hostosconfigurationmodules"
in API group "kaas.mirantis.com" at the cluster scope: access denied

Workaround:

Modify the global-admin role by adding a new entry with the following contents to the rules list:

kubectl edit clusterroles kaas-global-admin

- apiGroups: [kaas.mirantis.com]
  resources: [hostosconfigurationmodules]
  verbs: ['*']

For each Container Cloud project, modify the kaas-operator role by adding a new entry with the following contents to the rules list:

kubectl -n <projectName> edit roles kaas-operator

- apiGroups: [kaas.mirantis.com]
  resources: [hostosconfigurations]
  verbs: ['*']

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[41305] DHCP responses are lost between dnsmasq and dhcp-relay pods¶

Fixed in 2.28.0 (17.3.0 and 16.3.0)

The issue was observed on environments having management cluster nodes configured with a single L2 segment used for all network traffic (PXE and LCM/management networks).

To verify whether the cluster is affected:

Verify whether the dnsmasq and dhcp-relay pods run on the same node in the management cluster:

kubectl -n kaas get pods -o wide| grep -e "dhcp\|dnsmasq"

Example of system response:

dhcp-relay-7d85f75f76-5vdw2   2/2   Running   2 (36h ago)   36h   10.10.0.122     kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>
dnsmasq-8f4b484b4-slhbd       5/5   Running   1 (36h ago)   36h   10.233.123.75   kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>

If this is the case, proceed to the workaround below.

Workaround:

Make sure that at least two management cluster nodes are schedulable:

kubectl get node

Example of a positive system response:

NAME                                             STATUS   ROLES    AGE   VERSION
kaas-node-bcedb87b-b3ce-46a4-a4ca-ea3068689e40   Ready    master   37h   v1.27.10-mirantis-1
kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   Ready    master   37h   v1.27.10-mirantis-1
kaas-node-ad5a6f51-b98f-43c3-91d5-55fed3d0ff21   Ready    master   37h   v1.27.10-mirantis-1

Delete the dhcp-relay pod:

kubectl -n kaas delete pod <dhcp-relay-xxxxx>

Verify that the dnsmasq and dhcp-relay pods are scheduled into different nodes:

kubectl -n kaas get pods -o wide| grep -e "dhcp\|dnsmasq"

Example of a positive system response:

dhcp-relay-7d85f75f76-rkv03   2/2   Running   0             49s   10.10.0.121     kaas-node-bcedb87b-b3ce-46a4-a4ca-ea3068689e40   <none>   <none>
dnsmasq-8f4b484b4-slhbd       5/5   Running   1 (37h ago)   37h   10.233.123.75   kaas-node-8a24b81c-76d0-4d4c-8421-962bd39df5ad   <none>   <none>

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

vSphere¶

[40747] Unsupported Cluster release is available for managed cluster deployment¶

Fixed in 2.26.1

The Cluster release 16.0.0, which is not supported for greenfield vSphere-based deployments, is still available in the drop-down menu of the cluster creation window in the Container Cloud web UI.

Do not select this Cluster release to prevent deployment failures. Use the latest supported version instead.

LCM¶

[41540] LCM Agent cannot grab storage information on a host¶

Fixed in 17.1.5 and 16.1.5

{"level":"error","ts":"2024-05-02T12:26:10Z","logger":"agent", \
"msg":"get hardware details", \
"host":"kaas-node-548b2861-aed0-41c9-8ff2-10c5476b000b", \
"error":"new storage info: get disk info \"nvme0c0n1\": \
invoke command: exit status 1","errorVerbose":"exit status 1

As a workaround, on the affected node, create a symlink for any device indicated in lcm-agent logs. For example:

ln -sfn /dev/nvme0n1 /dev/nvme0c0n1

[40036] Node is not removed from a cluster when its Machine is disabled¶

Fixed in 2.26.1 (17.1.1 and 16.1.1)

During the ClusterRelease update of a MOSK cluster, a node cannot be removed from the Kubernetes cluster if the related Machine object is disabled.

As a workaround, remove the finalizer from the affected Node object.

[39437] Failure to replace a master node on a Container Cloud cluster¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

During the replacement of a master node on a cluster of any type, the process may get stuck with Kubelet's NodeReady condition is Unknown in the machine status on the remaining master nodes.

As a workaround, log in on the affected node and run the following command:

docker restart ucp-kubelet

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

[30294] Replacement of a master node is stuck on the calico-node Pod start¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a master node on a cluster of any type, the calico-node Pod fails to start on a new node that has the same IP address as the node being replaced.

Workaround:

From a CLI with an MKE client bundle, create a shell alias to start calicoctl using the mirantis/ucp-dsinfo image:

Since MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/key.pem \
-e ETCD_CA_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/ca.pem \
-e ETCD_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/cert.pem \
-v /var/run/calico:/var/run/calico \
-v /var/lib/docker/volumes/ucp-kv-certs/_data:/var/lib/docker/volumes/ucp-kv-certs/_data:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl \
"

Before MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
-e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
-e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
-v /var/run/calico:/var/run/calico \
-v ucp-node-certs:/ucp-node-certs:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl --allow-version-mismatch \
"

In the above command, replace the following values with the corresponding settings of the affected cluster:

<etcdEndpoint> is the etcd endpoint defined in the Calico configuration file. For example, ETCD_ENDPOINTS=127.0.0.1:12378
<mkeVersion> is the MKE version installed on your cluster. For example, mirantis/ucp-dsinfo:3.5.7.

Verify the node list on the cluster:
```
kubectl get node
```
Compare this list with the node list in Calico to identify the old node:
```
calicoctl get node -o wide
```

Remove the old node from Calico:

calicoctl delete node kaas-node-<nodeID>

[5782] Manager machine fails to be deployed during node replacement¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a manager machine, the following problems may occur:

The system adds the node to Docker swarm but not to Kubernetes
The node Deployment gets stuck with failed RethinkDB health checks

Workaround:

Delete the failed node.
Wait for the MKE cluster to become healthy. To monitor the cluster status:
1. Log in to the MKE web UI as described in Connect to the Mirantis Kubernetes Engine web UI.
2. Monitor the cluster status as described in MKE Operations Guide: Monitor an MKE cluster with the MKE web UI.
Deploy a new node.

[5568] The calico-kube-controllers Pod fails to clean up resources¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During the unsafe or forced deletion of a manager machine running the calico-kube-controllers Pod in the kube-system namespace, the following issues occur:

The calico-kube-controllers Pod fails to clean up resources associated with the deleted node
The calico-node Pod may fail to start up on a newly created node if the machine is provisioned with the same IP address as the deleted machine had

As a workaround, before deletion of the node running the calico-kube-controllers Pod, cordon and drain the node:

kubectl cordon <nodeName>
kubectl drain <nodeName>

Ceph¶

[41819] Graceful cluster reboot is blocked by the Ceph ClusterWorkloadLocks¶

Fixed in 2.27.0 (17.2.0 and 16.2.0)

During graceful reboot of a cluster with Ceph enabled, the reboot is blocked with the following message in the MiraCephMaintenance object status:

message: ClusterMaintenanceRequest found, Ceph Cluster is not ready to upgrade,
 delaying cluster maintenance

As a workaround, add the following snippet to the cephFS section under metadataServer in the spec section of <kcc-name>.yaml in the Ceph cluster:

cephClusterSpec:
  sharedFilesystem:
    cephFS:
    - name: cephfs-store
      metadataServer:
        activeCount: 1
        healthCheck:
          livenessProbe:
            probe:
              failureThreshold: 5
              initialDelaySeconds: 30
              periodSeconds: 30
              successThreshold: 1
              timeoutSeconds: 5

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

StackLight¶

[44193] OpenSearch reaches 85% disk usage watermark affecting the cluster state¶

Fixed in 2.29.0 (17.4.0 and 16.4.0)

Caution

The issue and the provided workaround apply only for clusters on which OpenSearch and Prometheus utilize the same storage pool.

To verify that the cluster is affected:

Verify the result of the following formula:
```
0.8 × OpenSearch_PVC_Size_GB + Prometheus_PVC_Size_GB > 0.85 × Total_Storage_Capacity_GB
```
In the formula, define the following values:
OpenSearch_PVC_Size_GB
Derived from .values.elasticsearch.persistentVolumeUsableStorageSizeGB, defaulting to .values.elasticsearch.persistentVolumeClaimSize if unspecified. To obtain the OpenSearch PVC size:
kubectl -n <namespaceName> get cluster <clusterName> -o yaml |\ yq '.spec.providerSpec.value.helmReleases[] | select(.name == "stacklight") | .values.elasticsearch.persistentVolumeClaimSize '
Example of system response:
10000Gi
Prometheus_PVC_Size_GB
Sourced from .values.prometheusServer.persistentVolumeClaimSize. To obtain the Prometheus PVC size:
kubectl -n <namespaceName> get cluster <clusterName> -o yaml |\ yq '.spec.providerSpec.value.helmReleases[] | select(.name == "stacklight") | .values.prometheusServer.persistentVolumeClaimSize '
Example of system response:
4000Gi
Total_Storage_Capacity_GB
Total capacity of the OpenSearch PVCs. For LVP, the capacity of the storage pool. To obtain the total capacity:
kubectl get pvc -n stacklight -l app=opensearch-master \ -o custom-columns=NAME:.metadata.name,CAPACITY:.status.capacity.storage
The system response contains multiple outputs, one per opensearch-master node. Select the capacity for the affected node.
Note

Convert the values to GB if they are set in different units.

If the formula result is positive, it is an early indication that the cluster is affected.
Verify whether the OpenSearchClusterStatusWarning or OpenSearchClusterStatusCritical alert is firing. And if so, verify the following:
1. Log in to the OpenSearch web UI.
2. In Management -> Dev Tools, run the following command:
```
GET _cluster/allocation/explain
```
  The following system response indicates that the corresponding node is affected:
```
"explanation": "the node is above the low watermark cluster setting \
[cluster.routing.allocation.disk.watermark.low=85%], using more disk space \
than the maximum allowed [85.0%], actual free: [xx.xxx%]"
```
  Note
  
  The system response may contain even higher watermark percent than 85.0%, depending on the case.

Workaround:

Warning

The workaround implies adjustement of the retention threshold for OpenSearch. And depending on the new threshold, some old logs will be deleted.

Adjust or set .values.elasticsearch.persistentVolumeUsableStorageSizeGB to a lower value for the affection check formula to be non-positive. For configuration details, see MOSK Operations Guide: StackLight configuration parameters - OpenSearch.

Mirantis also recommends reserving some space for other PVCs using storage from the pool. Use the following formula to calculate the required space:
```
persistentVolumeUsableStorageSizeGB =
0.84 × ((1 - Reserved_Percentage - Filesystem_Reserve) ×
Total_Storage_Capacity_GB - Prometheus_PVC_Size_GB) /
0.8
```
In the formula, define the following values:
Reserved_Percentage
A user-defined variable that specifies what percentage of the total storage capacity should not be used by OpenSearch or Prometheus. This is used to reserve space for other components. It should be expressed as a decimal. For example, for 5% of reservation, Reserved_Percentage is 0.05. Mirantis recommends using 0.05 as a starting point.

Filesystem_Reserve
Percentage to deduct for filesystems that may reserve some portion of the available storage, which is marked as occupied. For example, for EXT4, it is 5% by default, so the value must be 0.05.

Prometheus_PVC_Size_GB
Sourced from .values.prometheusServer.persistentVolumeClaimSize.

Total_Storage_Capacity_GB
Total capacity of the OpenSearch PVCs. For LVP, the capacity of the storage pool. To obtain the total capacity:
kubectl get pvc -n stacklight -l app=opensearch-master \ -o custom-columns=NAME:.metadata.name,CAPACITY:.status.capacity.storage
The system response contains multiple outputs, one per opensearch-master node. Select the capacity for the affected node.
Note

Convert the values to GB if they are set in different units.

Calculation of above formula provides a maximum safe storage to allocate for .values.elasticsearch.persistentVolumeUsableStorageSizeGB. Use this formula as a reference for setting .values.elasticsearch.persistentVolumeUsableStorageSizeGB on a cluster.
Wait up to 15-20 mins for OpenSearch to perform the cleaning.
Verify that the cluster is not affected anymore using the procedure above.

[42304] Failure of shard relocation in the OpenSearch cluster¶

Fixed in 17.2.0, 16.2.0, 17.1.6, 16.1.6

{created_by_kind="StatefulSet",created_by_name="opensearch-master",namespace="stacklight"}

Caution

To verify that the initialization of shards is ongoing:

kubectl exec -it pod/opensearch-master-0 -n stacklight -c opensearch -- bash

curl "http://localhost:9200/_cat/shards" | grep INITIALIZING

Example of system response:

.ds-system-000072    2 r INITIALIZING    10.232.182.135 opensearch-master-1
.ds-system-000073    1 r INITIALIZING    10.232.7.145   opensearch-master-2
.ds-system-000073    2 r INITIALIZING    10.232.182.135 opensearch-master-1
.ds-audit-000001     2 r INITIALIZING    10.232.7.145   opensearch-master-2

You can additionally analyze the exact level of throttling and the current CPU usage on the Kubernetes Containers dashboard in Grafana.

Workaround:

Verify the currently configured CPU requests and limits for the opensearch containers:

kubectl -n stacklight get statefulset.apps/opensearch-master -o jsonpath="{.spec.template.spec.containers[?(@.name=='opensearch')].resources}"

Example of system response:

{"limits":{"cpu":"600m","memory":"8Gi"},"requests":{"cpu":"500m","memory":"6Gi"}}

In the example above, the CPU request is 500m and the CPU limit is 600m.

Increase the CPU limit to a reasonably high number.

For example, the default CPU limit for the clusters with the clusterSize:large parameter set was increased from 8000m to 12000m for StackLight in Container Cloud 2.27.0 (Cluster releases 17.2.0 and 16.2.0).

Note

For details, on the clusterSize parameter, see MOSK Operations Guide: StackLight configuration parameters - Cluster size.

If the defaults are already overridden on the affected cluster using the resourcesPerClusterSize or resources parameters as described in MOSK Operations Guide: StackLight configuration parameters - Resource limits, then the exact recommended number depends on the currently set limit.

Mirantis recommends increasing the limit by 50%. If it does not resolve the issue, another increase iteration will be required.
When you select the required CPU limit, increase it as described in MOSK Operations Guide: StackLight configuration parameters - Resource limits.

If the CPU limit for the opensearch component is already set, increase it in the Cluster object for the opensearch parameter. Otherwise, the default StackLight limit is used. In this case, increase the CPU limit for the opensearch component using the resources parameter.
Wait until all opensearch-master pods are recreated with the new CPU limits and become running and ready.

To verify the current CPU limit for every opensearch container in every opensearch-master pod separately:
```
kubectl -n stacklight get pod/opensearch-master-<podSuffixNumber> -o jsonpath="{.spec.containers[?(@.name=='opensearch')].resources}"
```
In the command above, replace <podSuffixNumber> with the name of the pod suffix. For example, pod/opensearch-master-0 or pod/opensearch-master-2.

Example of system response:
```
{"limits":{"cpu":"900m","memory":"8Gi"},"requests":{"cpu":"500m","memory":"6Gi"}}
```
The waiting time may take up to 20 minutes depending on the cluster size.

If the KubeContainersCPUThrottlingHigh alert is still firing, proceed with another iteration of the CPU limit increase.

[40020] Rollover policy update is not appllied to the current index¶

Fixed in 17.2.0, 16.2.0, 17.1.6, 16.1.6

While updating rollover_policy for the current system* and audit* data streams, the update is not applied to indices.

2024-05-31 13:16:04,459 ERROR   Failed to complete action: delete_indices.  <class 'curator.exceptions.FailedExecution'>: Exception encountered.  Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: RequestError(400, 'illegal_argument_exception', 'index [.ds-audit-000001] is the write index for data stream [audit] and cannot be deleted')

2024-05-31 13:16:04,459 ERROR   Failed to complete action: delete_indices.  <class 'curator.exceptions.FailedExecution'>: Exception encountered.  Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: RequestError(400, 'illegal_argument_exception', 'index [.ds-system-000001] is the write index for data stream [system] and cannot be deleted')

Note

Instead of .ds-audit-000001 or .ds-system-000001 index names, similar names can be present with the same prefix but different suffix numbers.

If the above mentioned alert and errors are present, an immediate action is required, because it indicates that the corresponding index size has already exceeded the space allocated for the index.

To verify that the cluster is affected:

Caution

Verify and apply the workaround to both index patterns, system and audit, separately.

If one of indices is affected, the second one is most likely affected as well. Although in rare cases, only one index may be affected.

kubectl exec -it pod/opensearch-master-0 -n stacklight -c opensearch -- bash

Verify that the rollover policy is present:
- system:
```
curl localhost:9200/_plugins/_ism/policies/system_rollover_policy
```
- audit:
```
curl localhost:9200/_plugins/_ism/policies/audit_rollover_policy
```
The cluster is affected if the rollover policy is missing. Otherwise, proceed to the following step.

Verify the system response from the previous step. For example:

{"_id":"system_rollover_policy","_version":7229,"_seq_no":42362,"_primary_term":28,"policy":{"policy_id":"system_rollover_policy","description":"system index rollover policy.","last_updated_time":1708505222430,"schema_version":19,"error_notification":null,"default_state":"rollover","states":[{"name":"rollover","actions":[{"retry":{"count":3,"backoff":"exponential","delay":"1m"},"rollover":{"min_size":"14746mb","copy_alias":false}}],"transitions":[]}],"ism_template":[{"index_patterns":["system*"],"priority":200,"last_updated_time":1708505222430}]}}

Verify and capture the following items separately for every policy:

The _seq_no and _primary_term values
The rollover policy threshold, which is defined in policy.states[0].actions[0].rollover.min_size

List indices:

system:

curl localhost:9200/_cat/indices | grep system

Example of system response:

[...]
green open .ds-system-000001   FjglnZlcTKKfKNbosaE9Aw 2 1 1998295  0   1gb 507.9mb

audit:

curl localhost:9200/_cat/indices | grep audit

Example of system response:

[...]
green open .ds-audit-000001   FjglnZlcTKKfKNbosaE9Aw 2 1 1998295  0   1gb 507.9mb

Select the index with the highest number and verify the rollover policy attached to the index:
- system:
```
curl localhost:9200/_plugins/_ism/explain/.ds-system-000001
```
- audit:
```
curl localhost:9200/_plugins/_ism/explain/.ds-audit-000001
```
- If the rollover policy is not attached, the cluster is affected.
- If the rollover policy is attached but _seq_no and _primary_term numbers do not match the previously captured ones, the cluster is affected.
- If the index size drastically exceeds the defined threshold of the rollover policy (which is the previously captured min_size), the cluster is most probably affected.

Workaround:

kubectl exec -it pod/opensearch-master-0 -n stacklight -c opensearch -- bash

If the policy is attached to the index but has different _seq_no and _primary_term, remove the policy from the index:

Note

Use the index with the highest number in the name, which was captured during verification procedure.
- system:
```
curl -XPOST localhost:9200/_plugins/_ism/remove/.ds-system-000001
```
- audit:
```
curl -XPOST localhost:9200/_plugins/_ism/remove/.ds-audit-000001
```

Re-add the policy:

system:

curl -XPOST -H "Content-type: application/json" localhost:9200/_plugins/_ism/add/system* -d'{"policy_id":"system_rollover_policy"}'

audit:

curl -XPOST -H "Content-type: application/json" localhost:9200/_plugins/_ism/add/audit* -d'{"policy_id":"audit_rollover_policy"}'

Perform again the last step of the cluster verification procedure provided above and make sure that the policy is attached to the index and has the same _seq_no and _primary_term.

If the index size drastically exceeds the defined threshold of the rollover policy (which is the previously captured min_size), wait up to 15 minutes and verify that the additional index is created with the consecutive number in the index name. For example:
- system: if you applied changes to .ds-system-000001, wait until .ds-system-000002 is created.
- audit: if you applied changes to .ds-audit-000001, wait until .ds-audit-000002 is created.
If such index is not created, escalate the issue to Mirantis support.

Container Cloud web UI¶

[41806] Configuration of a management cluster fails without Keycloak settings¶

Fixed in 17.1.4 and 16.1.4

During configuration of a management cluster settings using the Configure cluster web UI menu, updating the Keycloak Truststore settings is mandatory, despite being optional.

As a workaround, update the management cluster using the API or CLI.

Components versions¶

The following table lists the major components and their versions delivered in the Container Cloud 2.26.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

**Container Cloud release components versions**¶
Component	Application/Service	Version
Bare metal ^Updated	ambasador	1.39.13
	baremetal-dnsmasq	base-2-26-alpine-20240129134230
	baremetal-operator	base-2-26-alpine-20240129135007
	baremetal-provider	1.39.13
	bm-collective	base-2-26-alpine-20240129155244
	cluster-api-provider-baremetal	1.39.13
	ironic	yoga-jammy-20240108060019
	ironic-inspector	yoga-jammy-20240108060019
	ironic-prometheus-exporter	0.1-20240117102150
	kaas-ipam	base-2-26-alpine-20240129213142
	kubernetes-entrypoint	1.0.1-55b02f7-20231019172556
	mariadb	10.6.14-focal-20231127070342
	metallb-controller	0.13.12-31212f9e-amd64
	metallb-speaker	0.13.12-31212f9e-amd64
	syslog-ng	base-alpine-20240129163811
Container Cloud	admission-controller ^Updated	1.39.13
	agent-controller ^Updated	1.39.13
	byo-cluster-api-controller ^New	1.39.13
	byo-credentials-controller ^New	1.39.13
	ceph-kcc-controller ^Updated	1.39.13
	cert-manager-controller	1.11.0-5
	cinder-csi-plugin ^Updated	1.27.2-11
	client-certificate-controller ^Updated	1.39.13
	configuration-collector ^Updated	1.39.13
	csi-attacher ^Updated	4.2.0-4
	csi-node-driver-registrar ^Updated	2.7.0-4
	csi-provisioner ^Updated	3.4.1-4
	csi-resizer ^Updated	1.7.0-4
	csi-snapshotter ^Updated	6.2.1-mcc-3
	event-controller ^Updated	1.39.13
	frontend ^Updated	1.39.13
	golang	1.20.4-alpine3.17
	iam-controller ^Updated	1.39.13
	kaas-exporter ^Updated	1.39.13
	kproxy ^Updated	1.39.13
	lcm-controller ^Updated	1.39.13
	license-controller ^Updated	1.39.13
	livenessprobe ^Updated	2.9.0-4
	machinepool-controller ^Updated	1.38.17
	mcc-haproxy ^Updated	0.24.0-46-gdaf7dbc
	metrics-server ^Updated	0.6.3-6
	nginx ^Updated	1.39.13
	policy-controller ^New	1.39.13
	portforward-controller ^Updated	1.39.13
	proxy-controller ^Updated	1.39.13
	rbac-controller ^Updated	1.39.13
	registry ^Updated	2.8.1-9
	release-controller ^Updated	1.39.13
	rhellicense-controller ^Updated	1.39.13
	scope-controller ^Updated	1.39.13
	storage-discovery ^Updated	1.39.13
	user-controller ^Updated	1.39.13
IAM	iam ^Updated	1.39.13
	iam-controller ^Updated	1.39.13
	keycloak ^Removed	n/a
	mcc-keycloak ^New	23.0.3-1
OpenStack ^Updated	host-os-modules-controller ^New	1.39.13
	openstack-cloud-controller-manager	v1.27.2-12
	openstack-cluster-api-controller	1.39.13
	openstack-provider	1.39.13
	os-credentials-controller	1.39.13
VMware vSphere	mcc-keepalived ^Updated	0.24.0-46-gdaf7dbc
	squid-proxy	0.0.1-10-g24a0d69
	vsphere-cloud-controller-manager ^New	v1.27.0-5
	vsphere-cluster-api-controller ^Updated	1.39.13
	vsphere-credentials-controller ^Updated	1.39.13
	vsphere-csi-driver ^New	v3.0.2-1
	vsphere-csi-syncer ^New	v3.0.2-1
	vsphere-provider ^Updated	1.39.13
	vsphere-vm-template-controller ^Updated	1.39.13

Artifacts¶

This section lists the artifacts of components included in the Container Cloud release 2.26.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Bare metal artifacts¶
Artifact	Component	Path
Binaries ^Updated	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20240201183421
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20240201183421
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-146-1bd8e71.tgz
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.39.13.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.39.13.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.39.13.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.39.13.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.39.13.tgz
	local-volume-provisioner	https://binary.mirantis.com/core/helm/local-volume-provisioner-1.39.13.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.39.13.tgz
Docker images ^Updated	ambasador	mirantis.azurecr.io/core/external/nginx:1.39.13
	baremetal-dnsmasq	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-26-alpine-20240129134230
	baremetal-operator	mirantis.azurecr.io/bm/baremetal-operator:base-2-26-alpine-20240129135007
	bm-collective	mirantis.azurecr.io/bm/bm-collective:base-2-26-alpine-20240129155244
	cluster-api-provider-baremetal	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.39.13
	ironic	mirantis.azurecr.io/openstack/ironic:yoga-jammy-20240108060019
	ironic-inspector	mirantis.azurecr.io/openstack/ironic-inspector:yoga-jammy-20240108060019
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20240117102150
	kaas-ipam	mirantis.azurecr.io/bm/kaas-ipam:base-2-26-alpine-20240129213142
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-55b02f7-20231019172556
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.14-focal-20231127070342
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.24.0-46-gdaf7dbc
	metallb-controller	mirantis.azurecr.io/bm/metallb/controller:v0.13.12-31212f9e-amd64
	metallb-speaker	mirantis.azurecr.io/bm/metallb/speaker:v0.13.12-31212f9e-amd64
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20240129163811

Core artifacts¶

Core artifacts¶
Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.39.13.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.39.13.tgz
Helm charts	admission-controller ^Updated	https://binary.mirantis.com/core/helm/admission-controller-1.39.13.tgz
	agent-controller ^Updated	https://binary.mirantis.com/core/helm/agent-controller-1.39.13.tgz
	byo-credentials-controller ^New	https://binary.mirantis.com/core/helm/byo-credentials-controller-1.39.13.tgz
	byo-provider ^New	https://binary.mirantis.com/core/helm/byo-provider-1.39.13.tgz
	ceph-kcc-controller ^Updated	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.39.13.tgz
	cert-manager ^Updated	https://binary.mirantis.com/core/helm/cert-manager-1.39.13.tgz
	cinder-csi-plugin ^Updated	https://binary.mirantis.com/core/helm/cinder-csi-plugin-1.39.13.tgz
	client-certificate-controller ^Updated	https://binary.mirantis.com/core/helm/client-certificate-controller-1.39.13.tgz
	configuration-collector ^Updated	https://binary.mirantis.com/core/helm/configuration-collector-1.39.13.tgz
	event-controller ^Updated	https://binary.mirantis.com/core/helm/event-controller-1.39.13.tgz
	host-os-modules-controller ^New	https://binary.mirantis.com/core/helm/host-os-modules-controller-1.39.13.tgz
	iam-controller ^Updated	https://binary.mirantis.com/core/helm/iam-controller-1.39.13.tgz
	kaas-exporter ^Updated	https://binary.mirantis.com/core/helm/kaas-exporter-1.39.13.tgz
	kaas-public-api ^Updated	https://binary.mirantis.com/core/helm/kaas-public-api-1.39.13.tgz
	kaas-ui ^Updated	https://binary.mirantis.com/core/helm/kaas-ui-1.39.13.tgz
	lcm-controller ^Updated	https://binary.mirantis.com/core/helm/lcm-controller-1.39.13.tgz
	license-controller ^Updated	https://binary.mirantis.com/core/helm/license-controller-1.39.13.tgz
	machinepool-controller ^Updated	https://binary.mirantis.com/core/helm/machinepool-controller-1.39.13.tgz
	mcc-cache ^Updated	https://binary.mirantis.com/core/helm/mcc-cache-1.39.13.tgz
	mcc-cache-warmup ^Updated	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.39.13.tgz
	metrics-server ^Updated	https://binary.mirantis.com/core/helm/metrics-server-1.39.13.tgz
	openstack-cloud-controller-manager ^Updated	https://binary.mirantis.com/core/helm/openstack-cloud-controller-manager-1.39.13.tgz
	openstack-provider ^Updated	https://binary.mirantis.com/core/helm/openstack-provider-1.39.13.tgz
	os-credentials-controller ^Updated	https://binary.mirantis.com/core/helm/os-credentials-controller-1.39.13.tgz
	policy-controller ^New	https://binary.mirantis.com/core/helm/policy-controller-1.39.13.tgz
	portforward-controller ^Updated	https://binary.mirantis.com/core/helm/portforward-controller-1.39.13.tgz
	proxy-controller ^Updated	https://binary.mirantis.com/core/helm/proxy-controller-1.39.13.tgz
	rbac-controller ^Updated	https://binary.mirantis.com/core/helm/rbac-controller-1.39.13.tgz
	release-controller ^Updated	https://binary.mirantis.com/core/helm/release-controller-1.39.13.tgz
	rhellicense-controller ^Updated	https://binary.mirantis.com/core/helm/rhellicense-controller-1.39.13.tgz
	scope-controller ^Updated	https://binary.mirantis.com/core/helm/scope-controller-1.39.13.tgz
	squid-proxy ^Updated	https://binary.mirantis.com/core/helm/squid-proxy-1.39.13.tgz
	user-controller ^Updated	https://binary.mirantis.com/core/helm/user-controller-1.39.13.tgz
	vsphere-cloud-controller-manager ^New	https://binary.mirantis.com/core/helm/vsphere-cloud-controller-manager-1.39.13.tgz
	vsphere-credentials-controller ^Updated	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.39.13.tgz
	vsphere-csi-plugin ^New	https://binary.mirantis.com/core/helm/vsphere-csi-plugin-1.39.13.tgz
	vsphere-provider ^Updated	https://binary.mirantis.com/core/helm/vsphere-provider-1.39.13.tgz
	vsphere-vm-template-controller ^Updated	https://binary.mirantis.com/core/helm/vsphere-vm-template-controller-1.39.13.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.39.13
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.39.13
	byo-cluster-api-controller ^New	mirantis.azurecr.io/core/byo-cluster-api-controller:1.39.13
	byo-credentials-controller ^New	mirantis.azurecr.io/core/byo-credentials-controller:1.39.13
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.39.13
	cert-manager-controller ^Updated	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-5
	cinder-csi-plugin ^Updated	mirantis.azurecr.io/lcm/kubernetes/cinder-csi-plugin:v1.27.2-11
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.39.13
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.39.13
	csi-attacher ^Updated	mirantis.azurecr.io/lcm/k8scsi/csi-attacher:v4.2.0-4
	csi-node-driver-registrar ^Updated	mirantis.azurecr.io/lcm/k8scsi/csi-node-driver-registrar:v2.7.0-4
	csi-provisioner ^Updated	mirantis.azurecr.io/lcm/k8scsi/csi-provisioner:v3.4.1-4
	csi-resizer ^Updated	mirantis.azurecr.io/lcm/k8scsi/csi-resizer:v1.7.0-4
	csi-snapshotter ^Updated	mirantis.azurecr.io/lcm/k8scsi/csi-snapshotter:v6.2.1-mcc-3
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.39.13
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.39.13
	host-os-modules-controller ^New	mirantis.azurecr.io/core/host-os-modules-controller:1.39.13
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.39.13
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.39.13
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.39.13
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.39.13
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.39.13
	livenessprobe ^Updated	mirantis.azurecr.io/lcm/k8scsi/livenessprobe:v2.9.0-4
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.39.13
	mcc-haproxy ^Updated	mirantis.azurecr.io/lcm/mcc-haproxy:v0.24.0-46-gdaf7dbc
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.24.0-46-gdaf7dbc
	metrics-server ^Updated	mirantis.azurecr.io/lcm/metrics-server-amd64:v0.6.3-6
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.39.13
	openstack-cloud-controller-manager ^Updated	mirantis.azurecr.io/lcm/kubernetes/openstack-cloud-controller-manager:v1.27.2-12
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.39.13
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.39.13
	policy-controller ^New	mirantis.azurecr.io/core/policy-controller:1.39.13
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.39.13
	proxy-controller ^Updated	mirantis.azurecr.io/core/proxy-controller:1.39.13
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.39.13
	registry ^Updated	mirantis.azurecr.io/lcm/registry:v2.8.1-9
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.39.13
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.39.13
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.39.13
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-10-g24a0d69
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.39.13
	vsphere-cloud-controller-manager ^New	mirantis.azurecr.io/lcm/kubernetes/vsphere-cloud-controller-manager:v1.27.0-5
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.39.13
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.39.13
	vsphere-csi-driver ^New	mirantis.azurecr.io/lcm/kubernetes/vsphere-csi-driver:v3.0.2-1
	vsphere-csi-syncer ^New	mirantis.azurecr.io/lcm/kubernetes/vsphere-csi-syncer:v3.0.2-1
	vsphere-vm-template-controller ^Updated	mirantis.azurecr.io/core/vsphere-vm-template-controller:1.39.13

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts ^Updated	iam	https://binary.mirantis.com/core/helm/iam-1.39.13.tgz
Docker images	keycloak ^Removed	n/a
	kubectl ^New	mirantis.azurecr.io/stacklight/kubectl:1.22-20240105023016
	kubernetes-entrypoint ^Updated	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-55b02f7-20231019172556
	mariadb ^Updated	mirantis.azurecr.io/general/mariadb:10.6.14-focal-20231127070342
	mcc-keycloak ^New	mirantis.azurecr.io/iam/mcc-keycloak:23.0.3-1

Security notes¶

The table below includes the total numbers of addressed unique and common vulnerabilities and exposures (CVE) by product component since the 2.25.4 patch release. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	Critical	High	Total
Ceph	Unique	0	2	2
Ceph	Common	0	6	6
Kaas core	Unique	0	7	7
Kaas core	Common	0	8	8
StackLight	Unique	3	7	10
StackLight	Common	5	19	24

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 24.1: Security notes.

Update notes¶

This section describes the specific actions you as a cloud operator need to complete before or after your Container Cloud cluster update to the Cluster releases 17.1.0 or 16.1.0.

Consider this information as a supplement to the generic update procedures published in Operations Guide: Automatic upgrade of a management cluster and Update a managed cluster.

Pre-update actions¶

Unblock cluster update by removing any pinned product artifacts¶

If any pinned product artifacts are present in the Cluster object of a management or managed cluster, the update will be blocked by the Admission Controller with the invalid HelmReleases configuration error until such artifacts are removed. The update process does not start and any changes in the Cluster object are blocked by the Admission Controller except the removal of fields with pinned product artifacts.

Therefore, verify that the following sections of the Cluster objects do not contain any image-related (tag, name, pullPolicy, repository) and global values inside Helm releases:

.spec.providerSpec.value.helmReleases
.spec.providerSpec.value.kaas.management.helmReleases
.spec.providerSpec.value.regionalHelmReleases
.spec.providerSpec.value.regional

For example, a cluster configuration that contains the following highlighted lines will be blocked until you remove them:

- name: kaas-ipam
          values:
            kaas_ipam:
              image:
                tag: base-focal-20230127092754
              exampleKey: exampleValue

- name: kaas-ipam
          values:
            global:
              anyKey: anyValue
            kaas_ipam:
              image:
                tag: base-focal-20230127092754
              exampleKey: exampleValue

The custom pinned product artifacts are inspected and blocked by the Admission Controller to ensure that Container Cloud clusters remain consistently updated with the latest security fixes and product improvements

Note

Update queries for custom log-based metrics in StackLight¶

Container Cloud 2.26.0 introduces reorganized and significantly improved StackLight logging pipeline. It involves changes in queries implemented in the scope of the logging.metricQueries feature designed for creation of custom log-based metrics. For the procedure, see StackLight operations: Create logs-based metrics.

If you already have some custom log-based metrics:

Before the cluster update, save existing queries.
After the cluster update, update the queries according to the changes implemented in the scope of the logging.metricQueries feature.

These steps prevent failures of queries containing fields that are renamed or removed in Container Cloud 2.26.0.

Post-update actions¶

Update bird configuration on BGP-enabled bare metal clusters¶

Container Cloud 2.26.0 introduces the bird daemon update from v1.6.8 to v2.0.7 on master nodes if BGP is used for BGP announcement of the cluster API load balancer address.

Review and adjust the storage parameters for OpenSearch¶

To prevent underused or overused storage space, review your storage space parameters for OpenSearch on the StackLight cluster:

Review the value of elasticsearch.persistentVolumeClaimSize and the real storage available on volumes.
Decide whether you have to additionally set elasticsearch.persistentVolumeUsableStorageSizeGB.

For both parameters description, see MOSK Operations Guide: StackLight configuration parameters - OpenSearch.

See also

2.25.4¶

The Container Cloud patch release 2.25.4, which is based on the 2.25.0 major release, provides the following updates:

Support for the patch Cluster releases 16.0.4 and 17.0.4 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 23.3.4.
Security fixes for CVEs in images.

This patch release also supports the latest major Cluster releases 17.0.0 and 16.0.0. And it does not support greenfield deployments based on deprecated Cluster releases. Use the latest available Cluster release instead.

For main deliverables of the parent Container Cloud release of 2.25.4, refer to 2.25.0.

Artifacts¶

This section lists the artifacts of components included in the Container Cloud patch release 2.25.4. For artifacts of the Cluster releases introduced in 2.25.4, see patch Cluster releases 17.0.4 and 16.0.4.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20231012141354
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20231012141354
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-113-4f8b843.tgz
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.38.33.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.38.33.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.38.33.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.38.33.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.38.33.tgz
	local-volume-provisioner	https://binary.mirantis.com/core/helm/local-volume-provisioner-1.38.33.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.38.33.tgz
Docker images	ambasador ^Updated	mirantis.azurecr.io/core/external/nginx:1.38.33
	baremetal-dnsmasq	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-25-alpine-20231128145936
	baremetal-operator	mirantis.azurecr.io/bm/baremetal-operator:base-2-25-alpine-20231204121500
	bm-collective	mirantis.azurecr.io/bm/bm-collective:base-2-25-alpine-20231121115652
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.38.33
	ironic	mirantis.azurecr.io/openstack/ironic:yoga-jammy-20231204153029
	ironic-inspector	mirantis.azurecr.io/openstack/ironic-inspector:yoga-jammy-20231204153029
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20231204142028
	kaas-ipam	mirantis.azurecr.io/bm/kaas-ipam:base-2-25-alpine-20231121164200
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-55b02f7-20231019172556
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.14-focal-20231127070342
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.23.0-88-g35be0fc
	metallb-controller	mirantis.azurecr.io/bm/metallb/controller:v0.13.9-ef4faae9-amd64
	metallb-speaker	mirantis.azurecr.io/bm/metallb/speaker:v0.13.9-ef4faae9-amd64
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20231121121917

Core artifacts¶

Artifact	Component	Paths
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.38.33.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.38.33.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.38.33.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.38.33.tgz
	byo-credentials-controller	https://binary.mirantis.com/core/helm/byo-credentials-controller-1.38.33.tgz
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.38.33.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.38.33.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.38.33.tgz
	cinder-csi-plugin	https://binary.mirantis.com/core/helm/cinder-csi-plugin-1.38.33.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.38.33.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.38.33.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.38.33.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.38.33.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.38.33.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.38.33.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.38.33.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.38.33.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.38.33.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.38.33.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.38.33.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.38.33.tgz
	metrics-server	https://binary.mirantis.com/core/helm/metrics-server-1.38.33.tgz
	openstack-cloud-controller-manager	https://binary.mirantis.com/core/helm/openstack-cloud-controller-manager-1.38.33.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.38.33.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.38.33.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.38.33.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.38.33.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.38.33.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.38.33.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.38.33.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.38.33.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.38.33.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.38.33.tgz
	vsphere-cloud-controller-manager	https://binary.mirantis.com/core/helm/vsphere-cloud-controller-manager-1.38.33.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.38.33.tgz
	vsphere-csi-plugin	https://binary.mirantis.com/core/helm/vsphere-csi-plugin-1.38.33.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.38.33.tgz
	vsphere-vm-template-controller	https://binary.mirantis.com/core/helm/vsphere-vm-template-controller-1.38.33.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.38.33
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.38.33
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.38.33
	byo-credentials-controller ^Updated	mirantis.azurecr.io/core/byo-credentials-controller:1.38.33
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.38.33
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-5
	cinder-csi-plugin	mirantis.azurecr.io/lcm/kubernetes/cinder-csi-plugin:v1.27.2-11
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.38.33
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.38.33
	csi-attacher	mirantis.azurecr.io/lcm/k8scsi/csi-attacher:v4.2.0-4
	csi-node-driver-registrar	mirantis.azurecr.io/lcm/k8scsi/csi-node-driver-registrar:v2.7.0-4
	csi-provisioner	mirantis.azurecr.io/lcm/k8scsi/csi-provisioner:v3.4.1-4
	csi-resizer	mirantis.azurecr.io/lcm/k8scsi/csi-resizer:v1.7.0-4
	csi-snapshotter	mirantis.azurecr.io/lcm/k8scsi/csi-snapshotter:v6.2.1-mcc-3
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.38.33
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.38.33
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.38.33
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.38.33
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.38.33
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.38.33
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.38.33
	livenessprobe	mirantis.azurecr.io/lcm/k8scsi/livenessprobe:v2.9.0-4
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.38.33
	mcc-haproxy ^Updated	mirantis.azurecr.io/lcm/mcc-haproxy:v0.23.0-88-g35be0fc
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.23.0-88-g35be0fc
	metrics-server	mirantis.azurecr.io/lcm/metrics-server-amd64:v0.6.3-6
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.38.33
	openstack-cloud-controller-manager	mirantis.azurecr.io/lcm/kubernetes/openstack-cloud-controller-manager:v1.27.2-12
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.38.33
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.38.33
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.38.33
	proxy-controller ^Updated	mirantis.azurecr.io/core/proxy-controller:1.38.33
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.38.33
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-7
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.38.33
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.38.33
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.38.33
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-10-g24a0d69
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.38.33
	vsphere-cloud-controller-manager	mirantis.azurecr.io/lcm/kubernetes/vsphere-cloud-controller-manager:v1.27.0-5
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.38.33
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.38.33
	vsphere-csi-driver	mirantis.azurecr.io/lcm/kubernetes/vsphere-csi-driver:v3.0.2-1
	vsphere-csi-syncer	mirantis.azurecr.io/lcm/kubernetes/vsphere-csi-syncer:v3.0.2-1
	vsphere-vm-template-controller ^Updated	mirantis.azurecr.io/core/vsphere-vm-template-controller:1.38.33

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts ^Updated	iam	https://binary.mirantis.com/iam/helm/iam-2.6.4.tgz
Docker images	kubectl ^Updated	mirantis.azurecr.io/stacklight/kubectl:1.22-20231208023019
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-55b02f7-20231019172556
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.14-focal-20231127070342
	mcc-keycloak	mirantis.azurecr.io/iam/mcc-keycloak:22.0.5-1

Security notes¶

The table below includes the total numbers of addressed unique and common CVEs in images by product component since the Container Cloud 2.25.3 patch release. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	High	Total
Ceph	Unique	1	1
Ceph	Common	5	5
Kaas core	Unique	1	1
Kaas core	Common	1	1
StackLight	Unique	3	3
StackLight	Common	9	9

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 23.3.4: Security notes.

Addressed issues¶

The following issues have been addressed in the Container Cloud patch release 2.25.4 along with the patch Cluster releases 17.0.4 and 16.0.4.

[38259] Fixed the issue causing the failure to attach an existing MKE cluster to a Container Cloud management cluster. The issue was related to byo-provider and prevented the attachment of MKE clusters having less than three manager nodes and two worker nodes.
[38399] Fixed the issue causing the failure to deploy a management cluster in the offline mode due to the issue in the setup script.

See also

Patch releases

See also

Releases delivered in 2023¶

This section contains historical information on the unsupported Container Cloud releases delivered in 2023. For the latest supported Container Cloud release, see Container Cloud releases.

Unsupported Container Cloud releases 2023¶
Version	Release date	Summary
2.25.3	Dec 18, 2023	Container Cloud 2.25.3 is the third patch release of the 2.25.x and MOSK 23.3.x release series that introduces the following updates: Support for MKE 3.7.3 Patch Cluster release 17.0.3 for MOSK 23.3.3 Patch Cluster release 16.0.3 Security fixes for CVEs in images
2.25.2	Dec 05, 2023	Container Cloud 2.25.2 is the second patch release of the 2.25.x and MOSK 23.3.x release series that introduces the following updates: Support for attachment of non Container Cloud based MKE clusters to vSphere-based management clusters Patch Cluster release 17.0.2 for MOSK 23.3.2 Patch Cluster release 16.0.2 Security fixes for CVEs in images
2.25.1	Nov 27, 2023	Container Cloud 2.25.1 is the first patch release of the 2.25.x and MOSK 23.3.x release series that introduces the following updates: MKE: Support for MKE 3.7.2 Amendments for MKE configuration managed by Container Cloud vSphere: Switch to an external vSphere cloud controller manager Mandatory MKE upgrade from 3.6 from 3.7 StackLight: Kubernetes Network Policies MKE benchmark compliance Patch Cluster release 17.0.1 for MOSK 23.3.1 Patch Cluster release 16.0.1 Security fixes for CVEs in images
2.25.0	Nov 06, 2023	Container Cloud Bootstrap v2 Support for MKE 3.7.1 and MCR 23.0.7 General availability for RHEL 8.7 on vSphere-based clusters Automatic cleanup of old Ubuntu kernel packages Configuration of a custom OIDC provider for MKE on managed clusters General availability for graceful machine deletion Bare metal provider: General availability for `MetalLBConfigTemplate` and `MetalLBConfig` objects Manual IP address allocation for bare metal hosts during PXE provisioning Ceph: Addressing storage devices using `by-id` identifiers Verbose Ceph cluster status in the `KaaSCephCluster.status` specification Detailed view of a Ceph cluster summary in web UI StackLight: Fluentd log forwarding to Splunk Ceph monitoring improvements Optimization of StackLight `NodeDown` alerts OpenSearch performance optimization Documentation: Export data from Table panels of Grafana dashboards to CSV Container Cloud web UI: Status of infrastructure health for bare metal and OpenStack providers Parallel update of worker nodes Graceful machine deletion
2.24.5	Sep 26, 2023	Container Cloud 2.24.4 is the third patch release of the 2.24.x and MOSK 23.2.x release series that introduces the following updates: Patch Cluster release 15.0.4 for MOSK 23.2.3 Patch Cluster release 14.0.4 Security fixes for CVEs of Critical and High severity
2.24.4	Sep 14, 2023	Container Cloud 2.24.4 is the second patch release of the 2.24.x and MOSK 23.2.x release series that introduces the following updates: Patch Cluster release 15.0.3 for MOSK 23.2.2 Patch Cluster release 14.0.3 Multi-rack topology for bare metal managed clusters Configuration of the etcd storage quota Security fixes for CVEs of Critical and High severity
2.24.3	Aug 29, 2023	Container Cloud 2.24.3 is the first patch release of the 2.24.x and MOSK 23.2.x release series that introduces the following updates: Patch Cluster release 15.0.2 for MOSK 23.2.1 Patch Cluster release 14.0.2 Support for MKE 3.6.6 and updated `docker-ee-cli` 20.10.18 for MCR 20.10.17 GA for TLS certificates configuration Security fixes for CVEs of High severity End of support for new deployments on deprecated major or patch Cluster releases For details, see Patch releases.
2.24.2	Aug 21, 2023	Based on 2.24.1, Container Cloud 2.24.2: Introduces the major Cluster release 15.0.1 that is based on 14.0.1 and supports Mirantis OpenStack for Kubernetes (MOSK) 23.2. Supports the Cluster release 14.0.1. The deprecated Cluster release 14.0.0 and the 12.7.x along with 11.7.x series are not supported for new deployments. Contains features and amendments of the parent releases 2.24.0 and 2.24.1.
2.24.1	Jul 27, 2023	Patch release containing hot fixes for the major Container Cloud release 2.24.0.
2.24.0	Jul 20, 2023	Support for MKE 3.6.5 and MCR 20.10.17 Bare metal: Automated upgrade of operating system on management and regional clusters Support for WireGuard Configuration of MTU size for Calico MetalLB configuration changes vSphere: Support for RHEL 8.7 MetalLB configuration changes OpenStack: Custom flavors for Octavia Deletion of persistent volumes during a cluster deletion IAM: Support for Keycloak Quarkus The admin role for management cluster Security: Support for auditd General availability for TLS certificates configuration LCM: Custom host names for cluster machines Cache warm-up for managed clusters Ceph: Automatic upgrade of Ceph from Pacific to Quincy Ceph non-admin client for a shared Ceph cluster Dropping of redundant components from management and regional clusters Documentation enhancements for Ceph OSDs StackLight: Major version update of OpenSearch and OpenSearch Dashboards from 1.3.7 to 2.7.0 Monitoring of network connectivity between Ceph nodes Improvements to StackLight alerting Performance tuning of Grafana dashboards Dropped and white-listed metrics Container Cloud web UI: Graceful cluster reboot Creation and deletion of bare metal host credentials Node labeling improvements
2.23.5	June 05, 2023	Container Cloud 2.23.5 is the fourth patch release of the 2.23.0 and 2.23.1 major releases that: Contains security fixes for critical and high CVEs Introduces the patch Cluster release 12.7.4 for MOSK 23.1.4 Introduces the patch Cluster release 11.7.4 Supports all major Cluster releases introduced in previous 2.23.x releases Does not support new deployments on deprecated major or patch Cluster releases For details, see Patch releases.
2.23.4	May 22, 2023	Container Cloud 2.23.4 is the third patch release of the 2.23.0 and 2.23.1 major releases that: Contains several addressed issues and security fixes for critical and high CVEs Introduces the patch Cluster release 12.7.3 for MOSK 23.1.3 Introduces the patch Cluster release 11.7.3 Supports all major Cluster releases introduced in previous 2.23.x releases Does not support new deployments on deprecated major or patch Cluster releases For details, see Patch releases.
2.23.3	May 04, 2023	Container Cloud 2.23.3 is the second patch release of the 2.23.0 and 2.23.1 major releases that: Contains security fixes for critical and high CVEs Introduces the patch Cluster release 12.7.2 for MOSK 23.1.2 Introduces the patch Cluster release 11.7.2 Supports all major Cluster releases introduced in previous 2.23.x releases Does not support new deployments on deprecated major or patch Cluster releases For details, see Patch releases.
2.23.2	Apr 20, 2023	Container Cloud 2.23.2 is the first patch release of the 2.23.0 and 2.23.1 major releases that: Contains security fixes for critical and high CVEs Introduces support for patch Cluster releases 12.7.1 or 11.7.1 Supports all major Cluster releases introduced and supported in the previous 2.23.x releases For details, see Patch releases.
2.23.1	Apr 04, 2023	Based on 2.23.0, Container Cloud 2.23.1: Introduces the Cluster release 12.7.0 that is based on 11.7.0 and supports Mirantis OpenStack for Kubernetes (MOSK) 23.1. Supports the Cluster release 11.7.0. The deprecated Cluster releases 12.5.0 and 11.6.0 are not supported for new deployments. Contains features and amendments of the parent releases 2.23.0 and 2.22.0.
2.23.0	Mar 07, 2023	MKE patch release update from 3.5.5 to 3.5.7 Automatic upgrade of Ceph from Octopus 15.2.17 to Pacific 16.2.11 Graceful cluster reboot using the `GracefulRebootRequest` CR Readiness fields for `Machine` and `Cluster` objects Deletion of persistent volumes during an OpenStack-based cluster deletion Option to disable time sync management Upgrade button for easy cluster update through the web UI Deployment of an Equinix Metal regional cluster with private networking on top of a public management cluster StackLight: HA setup for `iam-proxy` in StackLight Log forwarding to third-party systems using Fluentd plugins MCC Applications Performance Grafana dashboard PVC configuration for Reference Application
2.22.0	Jan 31, 2023	Custom network configuration for Equinix Metal managed clusters Custom TLS certificates for the StackLight `iam-proxy` endpoints Notification of a required reboot in the status of a bare metal machine Cluster deployment and update history objects Extended logging format for essential management cluster components StackLight: Bond interfaces monitoring Calculation of storage retention time Deployment of cAdvisor as a StackLight component Container Cloud web UI support for Reference Application Ceph: Two Ceph Managers by default for HA General availability of Ceph Shared File System Sharing Ceph between managed clusters or to an attached MKE cluster

2.25.3¶

The Container Cloud patch release 2.25.3, which is based on the 2.25.0 major release, provides the following updates:

Support for MKE 3.7.3. For details, see MKE documentation: Release Notes.
Support for the patch Cluster releases 16.0.3 and 17.0.3 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 23.3.3.
Security fixes for CVEs in images.

For main deliverables of the parent Container Cloud release of 2.25.3, refer to 2.25.0.

Artifacts¶

This section lists the artifacts of components included in the Container Cloud patch release 2.25.3. For artifacts of the Cluster releases introduced in 2.25.3, see patch Cluster releases 17.0.3 and 16.0.3.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20231012141354
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20231012141354
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-113-4f8b843.tgz
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.38.31.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.38.31.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.38.31.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.38.31.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.38.31.tgz
	local-volume-provisioner	https://binary.mirantis.com/core/helm/local-volume-provisioner-1.38.31.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.38.31.tgz
Docker images	ambasador ^Updated	mirantis.azurecr.io/core/external/nginx:1.38.31
	baremetal-dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-25-alpine-20231128145936
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-2-25-alpine-20231204121500
	bm-collective	mirantis.azurecr.io/bm/bm-collective:base-2-25-alpine-20231121115652
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.38.31
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:yoga-jammy-20231204153029
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:yoga-jammy-20231204153029
	ironic-prometheus-exporter ^Updated	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20231204142028
	kaas-ipam	mirantis.azurecr.io/bm/kaas-ipam:base-2-25-alpine-20231121164200
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-55b02f7-20231019172556
	mariadb ^Updated	mirantis.azurecr.io/general/mariadb:10.6.14-focal-20231127070342
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.23.0-87-gc9d7d3b
	metallb-controller ^Updated	mirantis.azurecr.io/bm/metallb/controller:v0.13.9-ef4faae9-amd64
	metallb-speaker ^Updated	mirantis.azurecr.io/bm/metallb/speaker:v0.13.9-ef4faae9-amd64
	syslog-ng ^Updated	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20231121121917

Core artifacts¶

Artifact	Component	Paths
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.38.31.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.38.31.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.38.31.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.38.31.tgz
	byo-credentials-controller	https://binary.mirantis.com/core/helm/byo-credentials-controller-1.38.31.tgz
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.38.31.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.38.31.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.38.31.tgz
	cinder-csi-plugin	https://binary.mirantis.com/core/helm/cinder-csi-plugin-1.38.31.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.38.31.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.38.31.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.38.31.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.38.31.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.38.31.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.38.31.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.38.31.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.38.31.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.38.31.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.38.31.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.38.31.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.38.31.tgz
	metrics-server	https://binary.mirantis.com/core/helm/metrics-server-1.38.31.tgz
	openstack-cloud-controller-manager	https://binary.mirantis.com/core/helm/openstack-cloud-controller-manager-1.38.31.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.38.31.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.38.31.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.38.31.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.38.31.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.38.31.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.38.31.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.38.31.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.38.31.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.38.31.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.38.31.tgz
	vsphere-cloud-controller-manager	https://binary.mirantis.com/core/helm/vsphere-cloud-controller-manager-1.38.31.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.38.31.tgz
	vsphere-csi-plugin	https://binary.mirantis.com/core/helm/vsphere-csi-plugin-1.38.31.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.38.31.tgz
	vsphere-vm-template-controller	https://binary.mirantis.com/core/helm/vsphere-vm-template-controller-1.38.31.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.38.31
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.38.31
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.38.31
	byo-credentials-controller ^Updated	mirantis.azurecr.io/core/byo-credentials-controller:1.38.31
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.38.31
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-5
	cinder-csi-plugin	mirantis.azurecr.io/lcm/kubernetes/cinder-csi-plugin:v1.27.2-11
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.38.31
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.38.31
	csi-attacher	mirantis.azurecr.io/lcm/k8scsi/csi-attacher:v4.2.0-4
	csi-node-driver-registrar	mirantis.azurecr.io/lcm/k8scsi/csi-node-driver-registrar:v2.7.0-4
	csi-provisioner	mirantis.azurecr.io/lcm/k8scsi/csi-provisioner:v3.4.1-4
	csi-resizer	mirantis.azurecr.io/lcm/k8scsi/csi-resizer:v1.7.0-4
	csi-snapshotter	mirantis.azurecr.io/lcm/k8scsi/csi-snapshotter:v6.2.1-mcc-3
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.38.31
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.38.31
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.38.31
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.38.31
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.38.31
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.38.31
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.38.31
	livenessprobe	mirantis.azurecr.io/lcm/k8scsi/livenessprobe:v2.9.0-4
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.38.31
	mcc-haproxy ^Updated	mirantis.azurecr.io/lcm/mcc-haproxy:v0.23.0-87-gc9d7d3b
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.23.0-87-gc9d7d3b
	metrics-server	mirantis.azurecr.io/lcm/metrics-server-amd64:v0.6.3-6
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.38.31
	openstack-cloud-controller-manager	mirantis.azurecr.io/lcm/kubernetes/openstack-cloud-controller-manager:v1.27.2-12
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.38.31
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.38.31
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.38.31
	proxy-controller ^Updated	mirantis.azurecr.io/core/proxy-controller:1.38.31
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.38.31
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-7
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.38.31
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.38.31
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.38.31
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-10-g24a0d69
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.38.31
	vsphere-cloud-controller-manager	mirantis.azurecr.io/lcm/kubernetes/vsphere-cloud-controller-manager:v1.27.0-5
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.38.31
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.38.31
	vsphere-csi-driver	mirantis.azurecr.io/lcm/kubernetes/vsphere-csi-driver:v3.0.2-1
	vsphere-csi-syncer	mirantis.azurecr.io/lcm/kubernetes/vsphere-csi-syncer:v3.0.2-1
	vsphere-vm-template-controller ^Updated	mirantis.azurecr.io/core/vsphere-vm-template-controller:1.38.31

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts ^Updated	iam	https://binary.mirantis.com/iam/helm/iam-2.6.3.tgz
Docker images	keycloak	n/a (replaced with `mcc-keycloak`)
	kubectl ^New	mirantis.azurecr.io/stacklight/kubectl:1.22-20231201023019
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-55b02f7-20231019172556
	mariadb ^Updated	mirantis.azurecr.io/general/mariadb:10.6.14-focal-20231127070342
	mcc-keycloak ^New	mirantis.azurecr.io/iam/mcc-keycloak:22.0.5-1

Security notes¶

The table below includes the total numbers of addressed unique and common CVEs in images by product component since the Container Cloud 2.25.2 patch release. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	Critical	High	Total
Ceph	Unique	0	1	1
Ceph	Common	0	3	3
KaaS core	Unique	2	9	11
KaaS core	Common	3	18	21
StackLight	Unique	1	18	19
StackLight	Common	1	52	53

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 23.3.3: Security notes.

Addressed issues¶

The following issues have been addressed in the Container Cloud patch release 2.25.3 along with the patch Cluster releases 17.0.3 and 16.0.3.

[37634][OpenStack] Fixed the issue with a management or managed cluster deployment or upgrade being blocked by all pods being stuck in the Pending state due to incorrect secrets being used to initialize the OpenStack external Cloud Provider Interface.
[37766][IAM] Fixed the issue with sign-in to the MKE web UI of the management cluster using the Sign in with External Provider option, which failed with the invalid parameter: redirect_uri error.

See also

Patch releases

2.25.2¶

The Container Cloud patch release 2.25.2, which is based on the 2.25.0 major release, provides the following updates:

Renewed support for attachment of MKE clusters that are not originally deployed by Container Cloud for vSphere-based management clusters.
Support for the patch Cluster releases 16.0.2 and 17.0.2 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 23.3.2.
Security fixes for CVEs in images.

This patch release also supports the latest major Cluster releases 17.0.0 and 16.0.0. And it does not support greenfield deployments based on deprecated Cluster releases 14.0.1, 15.0.1, 16.0.1, and 17.0.1. Use the latest available Cluster releases instead.

For main deliverables of the parent Container Cloud release of 2.25.2, refer to 2.25.0.

Artifacts¶

This section lists the artifacts of components included in the Container Cloud patch release 2.25.2. For artifacts of the Cluster releases introduced in 2.25.2, see patch Cluster releases 17.0.2 and 16.0.2.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20231012141354
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20231012141354
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-113-4f8b843.tgz
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.38.29.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.38.29.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.38.29.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.38.29.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.38.29.tgz
	local-volume-provisioner	https://binary.mirantis.com/core/helm/local-volume-provisioner-1.38.29.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.38.29.tgz
Docker images	ambasador ^Updated	mirantis.azurecr.io/core/external/nginx:1.38.29
	baremetal-dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-2-25-alpine-20231121112823
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-2-25-alpine-20231121112816
	bm-collective ^Updated	mirantis.azurecr.io/bm/bm-collective:base-2-25-alpine-20231121115652
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.38.29
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:yoga-jammy-20231120060019
	ironic-inspector	mirantis.azurecr.io/openstack/ironic-inspector:yoga-jammy-20231030060018
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20230912104602
	kaas-ipam ^Updated	mirantis.azurecr.io/bm/kaas-ipam:base-2-25-alpine-20231121164200
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-55b02f7-20231019172556
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.14-focal-20231024091216
	mcc-keepalived	mirantis.azurecr.io/docker.mirantis.net/lcm/mcc-keepalived:v0.23.0-84-g8d74d7c
	metallb-controller ^Updated	mirantis.azurecr.io/bm/metallb/controller:v0.13.9-ef4faae9-amd64
	metallb-speaker ^Updated	mirantis.azurecr.io/bm/metallb/speaker:v0.13.9-ef4faae9-amd64
	syslog-ng ^Updated	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20231121121917

Core artifacts¶

Artifact	Component	Paths
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.38.29.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.38.29.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.38.29.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.38.29.tgz
	byo-credentials-controller ^New	https://binary.mirantis.com/core/helm/byo-credentials-controller-1.38.29.tgz
	byo-provider ^New	https://binary.mirantis.com/core/helm/byo-provider-1.38.29.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.38.29.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.38.29.tgz
	cinder-csi-plugin	https://binary.mirantis.com/core/helm/cinder-csi-plugin-1.38.29.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.38.29.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.38.29.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.38.29.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.38.29.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.38.29.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.38.29.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.38.29.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.38.29.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.38.29.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.38.29.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.38.29.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.38.29.tgz
	metrics-server	https://binary.mirantis.com/core/helm/metrics-server-1.38.29.tgz
	openstack-cloud-controller-manager	https://binary.mirantis.com/core/helm/openstack-cloud-controller-manager-1.38.29.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.38.29.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.38.29.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.38.29.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.38.29.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.38.29.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.38.29.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.38.29.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.38.29.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.38.29.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.38.29.tgz
	vsphere-cloud-controller-manager	https://binary.mirantis.com/core/helm/vsphere-cloud-controller-manager-1.38.29.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.38.29.tgz
	vsphere-csi-plugin	https://binary.mirantis.com/core/helm/vsphere-csi-plugin-1.38.29.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.38.29.tgz
	vsphere-vm-template-controller	https://binary.mirantis.com/core/helm/vsphere-vm-template-controller-1.38.29.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.38.29
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.38.29
	byo-credentials-controller ^New	mirantis.azurecr.io/core/byo-credentials-controller:1.38.29
	byo-provider ^New	mirantis.azurecr.io/core/byo-provider:1.38.29
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.38.29
	cert-manager-controller ^Updated	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-5
	cinder-csi-plugin	mirantis.azurecr.io/lcm/kubernetes/cinder-csi-plugin:v1.27.2-11
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.38.29
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.38.29
	csi-attacher	mirantis.azurecr.io/lcm/k8scsi/csi-attacher:v4.2.0-4
	csi-node-driver-registrar	mirantis.azurecr.io/lcm/k8scsi/csi-node-driver-registrar:v2.7.0-4
	csi-provisioner	mirantis.azurecr.io/lcm/k8scsi/csi-provisioner:v3.4.1-4
	csi-resizer	mirantis.azurecr.io/lcm/k8scsi/csi-resizer:v1.7.0-4
	csi-snapshotter	mirantis.azurecr.io/lcm/k8scsi/csi-snapshotter:v6.2.1-mcc-3
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.38.29
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.38.29
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.38.29
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.38.29
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.38.29
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.38.29
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.38.29
	livenessprobe	mirantis.azurecr.io/lcm/k8scsi/livenessprobe:v2.9.0-4
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.38.29
	mcc-haproxy	mirantis.azurecr.io/lcm/mcc-haproxy:v0.23.0-84-g8d74d7c
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.23.0-84-g8d74d7c
	metrics-server ^Updated	mirantis.azurecr.io/lcm/metrics-server-amd64:v0.6.3-6
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.38.29
	openstack-cloud-controller-manager ^Updated	mirantis.azurecr.io/lcm/kubernetes/openstack-cloud-controller-manager:v1.27.2-12
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.38.29
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.38.29
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.38.29
	proxy-controller ^Updated	mirantis.azurecr.io/core/proxy-controller:1.38.29
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.38.29
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-7
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.38.29
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.38.29
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.38.29
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-10-g24a0d69
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.38.29
	vsphere-cloud-controller-manager ^Updated	mirantis.azurecr.io/lcm/kubernetes/vsphere-cloud-controller-manager:v1.27.0-5
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.38.29
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.38.29
	vsphere-csi-driver ^Updated	mirantis.azurecr.io/lcm/kubernetes/vsphere-csi-driver:v3.0.2-1
	vsphere-csi-syncer ^Updated	mirantis.azurecr.io/lcm/kubernetes/vsphere-csi-syncer:v3.0.2-1
	vsphere-vm-template-controller ^Updated	mirantis.azurecr.io/core/vsphere-vm-template-controller:1.38.29

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts	iam	https://binary.mirantis.com/iam/helm/iam-2.5.10.tgz
Docker images	keycloak	mirantis.azurecr.io/iam/keycloak:0.6.0-1
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-55b02f7-20231019172556
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.14-focal-20231024091216

Security notes¶

The table below includes the total numbers of addressed unique and common CVEs in images by product component since the Container Cloud 2.25.1 patch release. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Product component	CVE type	High	Total
Kaas core	Unique	6	6
Kaas core	Common	20	20
Ceph	Unique	2	2
Ceph	Common	6	6
StackLight	Unique	16	16
StackLight	Common	70	70

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 23.3.2: Security notes.

See also

Patch releases

2.25.1¶

The Container Cloud patch release 2.25.1, which is based on the 2.25.0 major release, provides the following updates:

Support for the patch Cluster releases 16.0.1 and 17.0.1 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 23.3.1.
Several product improvements. For details, see Enhancements.
Security fixes for CVEs in images.

This patch release also supports the latest major Cluster releases 17.0.0 and 16.0.0. And it does not support greenfield deployments based on deprecated Cluster releases 14.1.0, 14.0.1, and 15.0.1. Use the latest available Cluster releases instead.

For main deliverables of the parent Container Cloud release of 2.25.1, refer to 2.25.0.

Enhancements¶

This section outlines new features and enhancements introduced in the Container Cloud patch release 2.25.1 along with Cluster releases 17.0.1 and 16.0.1.

Support for MKE 3.7.2¶

Introduced support for Mirantis Kubernetes Engine (MKE) 3.7.2 on Container Cloud management and managed clusters. On existing managed clusters, MKE is updated to the latest supported version when you update your cluster to the patch Cluster release 17.0.1 or 16.0.1.

Learn more

MKE documentation: MKE release notes

MKE options managed by Container Cloud¶

To simplify MKE configuration through API, moved management of MKE parameters controlled by Container Cloud from lcm-ansible to lcm-controller. Now, Container Cloud overrides only a set of MKE configuration parameters that are automatically managed by Container Cloud.

Learn more

Reference Architecture: MKE options managed by Container Cloud

Improvements in the MKE benchmark compliance for StackLight¶

Analyzed and fixed the majority of failed compliance checks in the MKE benchmark compliance for StackLight. The following controls were analyzed:

Control ID	Control description	Analyzed item
5.2.7	Minimize the admission of containers with the `NET_RAW` capability	Containers with `NET_RAW` capability
5.2.6	Minimize the admission of root containers	Containers permitting root Containers with the `RunAsUser` root or root not set Containers with the `SYS_ADMIN` capability Container UID is a range of hosts

Kubernetes network policies in StackLight¶

Introduced Kubernetes network policies for all StackLight components. The feature is implemented using the networkPolicies parameter that is enabled by default.

The Kubernetes NetworkPolicy resource allows controlling network connections to and from Pods within a cluster. This enhances security by restricting communication from compromised Pod applications and provides transparency into how applications communicate with each other.

Learn more

External vSphere CCM with CSI supporting vSphere 6.7 on Kubernetes 1.27¶

Switched to the external vSphere cloud controller manager (CCM) that uses vSphere Container Storage Plug-in 3.0 for volume attachment. The feature implementation implies an automatic migration of PersistentVolume and PersistentVolumeClaim.

The external vSphere CCM supports vSphere 6.7 on Kubernetes 1.27 as compared to the in-tree vSphere CCM that does not support vSphere 6.7 since Kubernetes 1.25.

Important

The major Cluster release 14.1.0 is the last Cluster release for the vSphere provider based on MCR 20.10 and MKE 3.6.6 with Kubernetes 1.24. Therefore, Mirantis highly recommends updating your existing vSphere-based managed clusters to the Cluster release 16.0.1 that contains newer versions on MCR and MKE with Kubernetes. Otherwise, your management cluster upgrade to Container Cloud 2.25.2 will blocked.

For the update procedure, refer to Operations Guide: Update a patch Cluster release of a managed cluster.

Since Container Cloud 2.25.1, the major Cluster release 14.1.0 is deprecated. Greenfield vSphere-based deployments on this Cluster release are not supported. Use the patch Cluster release 16.0.1 for new deployments instead.

Learn more

VMware vSphere documentation: Migrating In-Tree vSphere Volumes to vSphere Container Storage Plug-in

Artifacts¶

This section lists the artifacts of components included in the Container Cloud patch release 2.25.1. For artifacts of the Cluster releases introduced in 2.25.1, see patch Cluster releases 17.0.1 and 16.0.1.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries ^Updated	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20231012141354
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20231012141354
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-113-4f8b843.tgz
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.38.22.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.38.22.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.38.22.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.38.22.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.38.22.tgz
	local-volume-provisioner	https://binary.mirantis.com/core/helm/local-volume-provisioner-1.38.22.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.38.22.tgz
Docker images ^Updated	ambasador	mirantis.azurecr.io/core/external/nginx:1.38.22
	baremetal-dnsmasq	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-alpine-20231030180650
	baremetal-operator	mirantis.azurecr.io/bm/baremetal-operator:base-alpine-20231101201729
	bm-collective	mirantis.azurecr.io/bm/bm-collective:base-alpine-20231027135748
	cluster-api-provider-baremetal	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.38.22
	ironic	mirantis.azurecr.io/openstack/ironic:yoga-jammy-20231030060018
	ironic-inspector	mirantis.azurecr.io/openstack/ironic-inspector:yoga-jammy-20231030060018
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20230912104602
	kaas-ipam	mirantis.azurecr.io/bm/kaas-ipam:base-alpine-20231027151726
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-55b02f7-20231019172556
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.14-focal-20231024091216
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.23.0-84-g8d74d7c
	metallb-controller	mirantis.azurecr.io/bm/metallb/controller:v0.13.9-fd3b03b0-amd64
	metallb-speaker	mirantis.azurecr.io/bm/metallb/speaker:v0.13.9-fd3b03b0-amd64
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-apline-20231030181839

Core artifacts¶

Artifact	Component	Paths
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com//core/binbootstrap-darwin-1.38.22.tgz
	bootstrap-linux	https://binary.mirantis.com//core/binbootstrap-linux-1.38.22.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.38.22.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.38.22.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.38.22.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.38.22.tgz
	cinder-csi-plugin	https://binary.mirantis.com/core/helm/cinder-csi-plugin-1.38.22.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.38.22.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.38.22.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.38.22.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.38.22.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.38.22.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.38.22.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.38.22.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.38.22.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.38.22.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.38.22.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.38.22.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.38.22.tgz
	metrics-server	https://binary.mirantis.com/core/helm/metrics-server-1.38.22.tgz
	openstack-cloud-controller-manager	https://binary.mirantis.com/core/helm/openstack-cloud-controller-manager-1.38.22.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.38.22.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.38.22.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.38.22.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.38.22.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.38.22.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.38.22.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.38.22.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.38.22.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.38.22.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.38.22.tgz
	vsphere-cloud-controller-manager ^New	https://binary.mirantis.com/core/helm/vsphere-cloud-controller-manager-1.38.22.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.38.22.tgz
	vsphere-csi-plugin ^New	https://binary.mirantis.com/core/helm/vsphere-csi-plugin-1.38.22.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.38.22.tgz
	vsphere-vm-template-controller	https://binary.mirantis.com/core/helm/vsphere-vm-template-controller-1.38.22.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.38.22
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.38.22
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.38.22
	cert-manager-controller ^Updated	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-4
	cinder-csi-plugin ^Updated	mirantis.azurecr.io/lcm/kubernetes/cinder-csi-plugin:v1.27.2-11
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.38.22
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.38.22
	csi-attacher ^Updated	mirantis.azurecr.io/lcm/k8scsi/csi-attacher:v4.2.0-4
	csi-node-driver-registrar ^Updated	mirantis.azurecr.io/lcm/k8scsi/csi-node-driver-registrar:v2.7.0-4
	csi-provisioner ^Updated	mirantis.azurecr.io/lcm/k8scsi/csi-provisioner:v3.4.1-4
	csi-resizer ^Updated	mirantis.azurecr.io/lcm/k8scsi/csi-resizer:v1.7.0-4
	csi-snapshotter ^Updated	mirantis.azurecr.io/lcm/k8scsi/csi-snapshotter:v6.2.1-mcc-3
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.38.22
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.38.22
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.38.22
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.38.22
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.38.22
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.38.22
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.38.22
	livenessprobe ^Updated	mirantis.azurecr.io/lcm/k8scsi/livenessprobe:v2.9.0-4
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.38.22
	mcc-haproxy ^Updated	mirantis.azurecr.io/lcm/mcc-haproxy:v0.23.0-84-g8d74d7c
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.23.0-84-g8d74d7c
	metrics-server ^Updated	mirantis.azurecr.io/lcm/metrics-server-amd64:v0.6.3-4
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.38.22
	openstack-cloud-controller-manager ^Updated	mirantis.azurecr.io/lcm/kubernetes/openstack-cloud-controller-manager:v1.27.2-11
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.38.22
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.38.22
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.38.22
	proxy-controller ^Updated	mirantis.azurecr.io/core/proxy-controller:1.38.22
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.38.22
	registry ^Updated	mirantis.azurecr.io/lcm/registry:v2.8.1-7
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.38.22
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.38.22
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.38.22
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-10-g24a0d69
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.38.22
	vsphere-cloud-controller-manager ^New	mirantis.azurecr.io/lcm/kubernetes/vsphere-cloud-controller-manager:v1.27.0-4
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.38.22
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.38.22
	vsphere-csi-driver ^New	mirantis.azurecr.io/core/external/vsphere-csi-driver:v3.0.2
	vsphere-csi-syncer ^New	mirantis.azurecr.io/core/external/vsphere-csi-syncer:v3.0.2
	vsphere-vm-template-controller ^Updated	mirantis.azurecr.io/core/vsphere-vm-template-controller:1.38.22

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts ^Updated	iam	https://binary.mirantis.com/iam/helm/iam-2.5.10.tgz
Docker images ^Updated	keycloak	mirantis.azurecr.io/iam/keycloak:0.6.0-1
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-55b02f7-20231019172556
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.14-focal-20231024091216

Security notes¶

The table below includes the total numbers of addressed unique and common CVEs in images by product component since the Container Cloud 2.25.0 major release. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Container Cloud component	CVE type	Critical	High	Total
Kaas core	Unique	0	12	12
Kaas core	Common	0	280	280
Ceph	Unique	0	8	8
Ceph	Common	0	41	41
StackLight	Unique	4	33	37
StackLight	Common	18	130	148

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 23.3.1: Security notes.

Addressed issues¶

The following issues have been addressed in the Container Cloud patch release 2.25.1 along with the patch Cluster releases 17.0.1 and 16.0.1.

[35426] [StackLight] Fixed the issue with the prometheus-libvirt-exporter Pod failing to reconnect to libvirt after the libvirt Pod recovery from a failure.
[35339] [LCM] Fixed the issue with the LCM Ansible task of copying kubectl from the ucp-hyperkube image failing if kubectl exec is in use, for example, during a management cluster upgrade.
[35089] [bare metal, Calico] Fixed the issue with arbitrary Kubernetes pods getting stuck in an error loop due to a failed Calico networking setup for that pod.
[33936] [bare metal, Calico] Fixed the issue with deletion failure of a controller node during machine replacement due to the upstream Calico issue.

See also

Patch releases

2.25.0¶

The Mirantis Container Cloud major release 2.25.0:

Introduces support for the Cluster release 17.0.0 that is based on the Cluster release 16.0.0 and represents Mirantis OpenStack for Kubernetes (MOSK) 23.3.
Introduces support for the Cluster release 16.0.0 that is based on Mirantis Container Runtime (MCR) 23.0.7 and Mirantis Kubernetes Engine (MKE) 3.7.1 with Kubernetes 1.27.
Introduces support for the Cluster release 14.1.0 that is dedicated for the vSphere provider only. This is the last Cluster release for the vSphere provider based on MKE 3.6.6 with Kubernetes 1.24.
Does not support greenfield deployments on deprecated Cluster releases of the 15.x and 14.x series. Use the latest available Cluster releases of the series instead.

Caution

Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.25.0.

Enhancements¶

This section outlines new features and enhancements introduced in the Container Cloud release 2.25.0. For the list of enhancements delivered with the Cluster releases introduced by Container Cloud 2.25.0, see 17.0.0, 16.0.0, and 14.1.0.

Container Cloud Bootstrap v2¶

Implemented Container Cloud Bootstrap v2 that provides an exceptional user experience to set up Container Cloud. With Bootstrap v2, you also gain access to a comprehensive and user-friendly web UI for the OpenStack and vSphere providers.

Bootstrap v2 empowers you to effortlessly provision management clusters before deployment, while benefiting from a streamlined process that isolates each step. This approach not only simplifies the bootstrap process but also enhances troubleshooting capabilities for addressing any potential intermediate failures.

Note

The Bootstrap web UI support for the bare metal provider will be added in one of the following Container Cloud releases.

Learn more

Deployment Guide: Deploy Container Cloud using Boostrap v2

General availability for ‘MetalLBConfigTemplate’ and ‘MetalLBConfig’ objects¶

Completed development of the MetalLB configuration related to address allocation and announcement for load-balanced services using the MetalLBConfigTemplate object for bare metal and the MetalLBConfig object for vSphere. Container Cloud uses these objects in default templates as recommended during creation of a management or managed cluster.

At the same time, removed the possibility to use the deprecated options, such as configInline value of the MetalLB chart and the use of Subnet objects without new MetalLBConfigTemplate and MetalLBConfig objects.

Automated migration, which applied to these deprecated options during creation of clusters of any type or cluster update to Container Cloud 2.24.x, is removed automatically during your management cluster upgrade to Container Cloud 2.25.0. After that, any changes in MetalLB configuration related to address allocation and announcement for load-balanced services will be applied using the MetalLBConfig, MetalLBConfigTemplate, and Subnet objects only.

Learn more

Manual IP address allocation for bare metal hosts during PXE provisioning¶

Technology Preview

Implemented the following annotations for bare metal hosts that enable manual allocation of IP addresses during PXE provisioning on managed clusters:

host.dnsmasqs.metal3.io/address - assigns a specific IP address to a host
baremetalhost.metal3.io/detached - pauses automatic host management

These annotations are helpful if you have a limited amount of free and unused IP addresses for server provisioning. Using these annotations, you can manually create bare metal hosts one by one and provision servers in small, manually managed chunks.

Learn more

Status of infrastructure health for bare metal and OpenStack providers¶

Implemented the Infrastructure Status condition to monitor infrastructure readiness in the Container Cloud web UI during cluster deployment for bare metal and OpenStack providers. Readiness of the following components is monitored:

Bare metal: the MetalLBConfig object along with MetalLB and DHCP subnets
OpenStack: cluster network, routers, load balancers, and Bastion along with their ports and floating IPs

For the bare metal provider, also implemented the Infrastructure Status condition for machines to monitor readiness of the IPAMHost, L2Template, BareMetalHost, and BareMetalHostProfile objects associated with the machine.

General availability for RHEL 8.7 on vSphere-based clusters¶

Introduced general availability support for RHEL 8.7 on VMware vSphere-based clusters. You can install this operating system on any type of a Container Cloud cluster including the bootstrap node.

Note

RHEL 7.9 is not supported as the operating system for the bootstrap node.

Caution

A Container Cloud cluster based on mixed RHEL versions, such as RHEL 7.9 and 8.7, is not supported.

Automatic cleanup of old Ubuntu kernel packages¶

Implemented automatic cleanup of old Ubuntu kernel and other unnecessary system packages. During cleanup, Container Cloud keeps two most recent kernel versions, which is the default behavior of the Ubuntu apt autoremove command.

Mirantis recommends keeping two kernel versions with the previous kernel version as a fallback option in the event that the current kernel may become unstable at any time. However, if you absolutely require leaving only the latest version of kernel packages, you can use the cleanup-kernel-packages script after considering all possible risks.

Learn more

Operations Guide: Remove old Ubuntu kernel packages

Configuration of a custom OIDC provider for MKE on managed clusters¶

Implemented the ability to configure a custom OpenID Connect (OIDC) provider for MKE on managed clusters using the ClusterOIDCConfiguration custom resource. Using this resource, you can add your own OIDC provider configuration to authenticate user requests to Kubernetes.

Note

For OpenStack and StackLight, Container Cloud supports only Keycloak, which is configured on the management cluster, as the OIDC provider.

The admin role for management cluster¶

Implemented the management-admin OIDC role to grant full admin access specifically to a management cluster. This role enables the user to manage Pods and all other resources of the cluster, for example, for debugging purposes.

Learn more

General availability for graceful machine deletion¶

Introduced general availability support for graceful machine deletion with a safe cleanup of node resources:

Changed the default deletion policy from unsafe to graceful for machine deletion using the Container Cloud API.

Using the deletionPolicy: graceful parameter in the providerSpec.value section of the Machine object, the cloud provider controller prepares a machine for deletion by cordoning, draining, and removing the related node from Docker Swarm. If required, you can abort a machine deletion when using deletionPolicy: graceful, but only before the related node is removed from Docker Swarm.
Implemented the following machine deletion methods in the Container Cloud web UI: Graceful, Unsafe, Forced.
Added support for deletion of manager machines, which is intended only for replacement or recovery of failed nodes, for MOSK-based clusters using either of deletion policies mentioned above.

Learn more

General availability for parallel update of worker nodes¶

Completed development of the parallel update of worker nodes during cluster update by implementing the ability to configure the required options using the Container Cloud web UI. Parallelizing of node update operations significantly optimizes the update efficiency of large clusters.

The following options are added to the Create Cluster window:

Parallel Upgrade Of Worker Machines that sets the maximum number of worker nodes to update simultaneously
Parallel Preparation For Upgrade Of Worker Machines that sets the maximum number of worker nodes for which new artifacts are downloaded at a given moment of time

Learn more

Operations Guide: Configure the parallel update of worker nodes using web UI

Addressed issues¶

The following issues have been addressed in the Mirantis Container Cloud release 2.25.0 along with the Cluster releases 17.0.0, 16.0.0, and 14.1.0.

Note

This section provides descriptions of issues addressed since the last Container Cloud patch release 2.24.5.

For details on addressed issues in earlier patch releases since 2.24.0, which are also included into the major release 2.25.0, refer to 2.24.x patch releases.

[34462] [BM] Fixed the issue with incorrect handling of the DHCP egress traffic by reconfiguring the external traffic policy for the dhcp-lb Kubernetes Service. For details about the issue, refer to the Kubernetes upstream bug.

On existing clusters with multiple L2 segments using DHCP relays on the border switches, in order to successfully provision new nodes or reprovision existing ones, manually point the DHCP relays on your network infrastructure to the new IP address of the dhcp-lb Service of the Container Cloud cluster.

To obtain the new IP address:
```
kubectl -n kaas get service dhcp-lb
```
[35429] [BM] Fixed the issue with the WireGuard interface not having the IPv4 address assigned. The fix implies automatic restart of the calico-node Pod to allocate the IPv4 address on the WireGuard interface.
[36131] [BM] Fixed the issue with IpamHost object changes not being propagated to LCMMachine during netplan configuration after cluster deployment.
[34657] [LCM] Fixed the issue with iam-keycloak Pods not starting after powering up master nodes and starting the Container Cloud upgrade right after.
[34750] [LCM] Fixed the issue with journald generating a lot of log messages that already exist in the auditd log due to enabled systemd-journald-audit.socket.
[35738] [StackLight] Fixed the issue with ucp-node-exporter being unable to bind the port 9100 with the ucp-node-exporter failing to start due to a conflict with the StackLight node-exporter binding the same port.

The resolution of the issue involves an automatic change of the port for the StackLight node-exporter from 9100 to 19100. No manual port update is required.

If your cluster uses a firewall, add an additional firewall rule that grants the same permissions to port 19100 as those currently assigned to port 9100 on all cluster nodes.
[34296] [StackLight] Fixed the issue with the CPU over-consumption by helm-controller leading to the KubeContainersCPUThrottlingHigh alert firing.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.25.0 including the Cluster releases 17.0.0, 16.0.0, and 14.1.0.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal¶

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[35089] Calico does not set up networking for a pod¶

Fixed in 17.0.1 and 16.0.1 for MKE 3.7.2

An arbitrary Kubernetes pod may get stuck in an error loop due to a failed Calico networking setup for that pod. The pod cannot access any network resources. The issue occurs more often during cluster upgrade or node replacement, but this can sometimes happen during the new deployment as well.

You may find the following log for the failed pod IP (for example, 10.233.121.132) in calico-node logs:

felix/route_table.go 898: Syncing routes: found unexpected route; ignoring due to grace period. dest=10.233.121.132/32 ifaceName="cali9731b965838" ifaceRegex="^cali." ipVersion=0x4 tableIndex=254
felix/route_table.go 898: Syncing routes: found unexpected route; ignoring due to grace period. dest=10.233.121.132/32 ifaceName="cali9731b965838" ifaceRegex="^cali." ipVersion=0x4 tableIndex=254
...
felix/route_table.go 902: Remove old route dest=10.233.121.132/32 ifaceName="cali9731b965838" ifaceRegex="^cali.*" ipVersion=0x4 routeProblems=[]string{"unexpected route"} tableIndex=254
felix/conntrack.go 90: Removing conntrack flows ip=10.233.121.132

The workaround is to manually restart the affected pod:

kubectl delete pod <failedPodID>

[33936] Deletion failure of a controller node during machine replacement¶

Fixed in 17.0.1 and 16.0.1 for MKE 3.7.2

Due to the upstream Calico issue, a controller node cannot be deleted if the calico-node Pod is stuck blocking node deletion. One of the symptoms is the following warning in the baremetal-operator logs:

Resolving dependency Service dhcp-lb in namespace kaas failed: \
the server was unable to return a response in the time allotted,\
but may still be processing the request (get endpoints dhcp-lb).

As a workaround, delete the Pod that is stuck to retrigger the node deletion.

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

OpenStack¶

[37634] Cluster deployment or upgrade is blocked by all pods in ‘Pending’ state¶

Fixed in 17.0.3 and 16.0.3

When using OpenStackCredential with a custom CACert, a management or managed cluster deployment or upgrade is blocked by all pods being stuck in the Pending state. The issue is caused by incorrect secrets being used to initialize the OpenStack external Cloud Provider Interface.

As a workaround, copy CACert from the OpenStackCredential object to openstack-ca-secret:

kubectl --kubeconfig <pathToFailedClusterKubeconfig> patch secret -n kube-system openstack-ca-secret -p '{"data":{"ca.pem":"'$(kubectl --kubeconfig <pathToManagementClusterKubeconfig> -n <affectedProjectName> get openstackcredentials <credentialsName> -o go-template="{{.spec.CACert}}")'"}}'

If the CACert from the OpenStackCredential is not base64-encoded:

kubectl --kubeconfig <pathToFailedClusterKubeconfig> patch secret -n kube-system openstack-ca-secret -p '{"data":{"ca.pem":"'$(kubectl --kubeconfig <pathToManagementClusterKubeconfig> -n <affectedProjectName> get openstackcredentials <credentialsName> -o go-template="{{.spec.CACert}}" | base64)'"}}'

In either command above, replace the following values:

<pathToFailedClusterKubeconfig> is the file path to the affected managed or management cluster kubeconfig.
<pathToManagementClusterKubeconfig> is the file path to the Container Cloud management cluster kubeconfig.
<affectedProjectName> is the Container Cloud project name containing the cluster with stuck pods. For a management cluster, the value is default.
<credentialsName> is the OpenStackCredential name used for the deployment.

IAM¶

[37766] Sign-in to the MKE web UI fails with ‘invalid parameter: redirect_uri’¶

Fixed in 17.0.3 and 16.0.3

A sign-in to the MKE web UI of the management cluster using the Sign in with External Provider option can fail with the invalid parameter: redirect_uri error.

Workaround:

Log in to the Keycloak admin console.
In the sidebar menu, switch to the IAM realm.
Navigate to Clients > kaas.
On the page, navigate to Seetings > Access settings > Valid redirect URIs.
Add https://<mgmt mke ip>:6443/* to the list of valid redirect URIs and click Save.
Refresh the browser window with the sign-in URI.

LCM¶

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

[32761] Node cleanup fails due to remaining devices¶

Fixed in 17.1.0 and 16.1.0

On MOSK clusters, the Ansible provisioner may hang in a loop while trying to remove LVM thin pool logical volumes (LVs) due to issues with volume detection before removal. The Ansible provisioner cannot remove LVM thin pool LVs correctly, so it consistently detects the same volumes whenever it scans disks, leading to a repetitive cleanup process.

The following symptoms mean that a cluster can be affected:

A node was configured to use thin pool LVs. For example, it had the OpenStack Cinder role in the past.
A bare metal node deployment flaps between provisioninig and deprovisioning states.
In the Ansible provisioner logs, the following example warnings are growing:
```
88621.log:7389:2023-06-22 16:30:45.109 88621 ERROR ansible.plugins.callback.ironic_log
[-] Ansible task clean : fail failed on node 14eb0dbc-c73a-4298-8912-4bb12340ff49:
{'msg': 'There are more devices to clean', '_ansible_no_log': None, 'changed': False}
```
Important

There are more devices to clean is a regular warning indicating some in-progress tasks. But if the number of such warnings is growing along with the node flapping between provisioninig and deprovisioning states, the cluster is highly likely affected by the issue.

As a workaround, erase disks manually using any preferred tool.

[30294] Replacement of a master node is stuck on the calico-node Pod start¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a master node on a cluster of any type, the calico-node Pod fails to start on a new node that has the same IP address as the node being replaced.

Workaround:

From a CLI with an MKE client bundle, create a shell alias to start calicoctl using the mirantis/ucp-dsinfo image:

Since MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/key.pem \
-e ETCD_CA_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/ca.pem \
-e ETCD_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/cert.pem \
-v /var/run/calico:/var/run/calico \
-v /var/lib/docker/volumes/ucp-kv-certs/_data:/var/lib/docker/volumes/ucp-kv-certs/_data:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl \
"

Before MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
-e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
-e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
-v /var/run/calico:/var/run/calico \
-v ucp-node-certs:/ucp-node-certs:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl --allow-version-mismatch \
"

In the above command, replace the following values with the corresponding settings of the affected cluster:

<etcdEndpoint> is the etcd endpoint defined in the Calico configuration file. For example, ETCD_ENDPOINTS=127.0.0.1:12378
<mkeVersion> is the MKE version installed on your cluster. For example, mirantis/ucp-dsinfo:3.5.7.

Verify the node list on the cluster:
```
kubectl get node
```
Compare this list with the node list in Calico to identify the old node:
```
calicoctl get node -o wide
```

Remove the old node from Calico:

calicoctl delete node kaas-node-<nodeID>

[5782] Manager machine fails to be deployed during node replacement¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a manager machine, the following problems may occur:

The system adds the node to Docker swarm but not to Kubernetes
The node Deployment gets stuck with failed RethinkDB health checks

Workaround:

Delete the failed node.
Wait for the MKE cluster to become healthy. To monitor the cluster status:
1. Log in to the MKE web UI as described in Connect to the Mirantis Kubernetes Engine web UI.
2. Monitor the cluster status as described in MKE Operations Guide: Monitor an MKE cluster with the MKE web UI.
Deploy a new node.

[5568] The calico-kube-controllers Pod fails to clean up resources¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During the unsafe or forced deletion of a manager machine running the calico-kube-controllers Pod in the kube-system namespace, the following issues occur:

The calico-kube-controllers Pod fails to clean up resources associated with the deleted node
The calico-node Pod may fail to start up on a newly created node if the machine is provisioned with the same IP address as the deleted machine had

As a workaround, before deletion of the node running the calico-kube-controllers Pod, cordon and drain the node:

kubectl cordon <nodeName>
kubectl drain <nodeName>

Ceph¶

[34820] The Ceph ‘rook-operator’ fails to connect to RGW on FIPS nodes¶

Fixed in 17.1.0 and 16.1.0

Due to the upstream Ceph issue, on clusters with the Federal Information Processing Standard (FIPS) mode enabled, the Ceph rook-operator fails to connect to Ceph RADOS Gateway (RGW) pods.

As a workaround, do not place Ceph RGW pods on nodes where FIPS mode is enabled.

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

Update¶

[37268] Container Cloud upgrade is blocked by a node in ‘Prepare’ or ‘Deploy’ state¶

Fixed in 17.1.0 and 16.1.0

Container Cloud upgrade may be blocked by a node being stuck in the Prepare or Deploy state with error processing package openssh-server. The issue is caused by customizations in /etc/ssh/sshd_config, such as additional Match statements. This file is managed by Container Cloud and must not be altered manually.

As a workaround, move customizations from sshd_config to a new file in the /etc/ssh/sshd_config.d/ directory.

[36928] The helm-controller Deployment is stuck during cluster update¶

During a cluster update, a Kubernetes helm-controller Deployment may get stuck in a restarting Pod loop with Terminating and Running states flapping. Other Deployment types may also be affected.

As a workaround, restart the Deployment that got stuck:

kubectl -n <affectedProjectName> get deploy <affectedDeployName> -o yaml

kubectl -n <affectedProjectName> scale deploy <affectedDeployName> --replicas 0

kubectl -n <affectedProjectName> scale deploy <affectedDeployName> --replicas <replicasNumber>

In the command above, replace the following values:

<affectedProjectName> is the Container Cloud project name containing the cluster with stuck Pods
<affectedDeployName> is the Deployment name that failed to run Pods in the specified project
<replicasNumber> is the original number of replicas for the Deployment that you can obtain using the get deploy command

[33438] ‘CalicoDataplaneFailuresHigh’ alert is firing during cluster update¶

During cluster update of a managed bare metal cluster, the false positive CalicoDataplaneFailuresHigh alert may be firing. Disregard this alert, which will disappear once cluster update succeeds.

The observed behavior is typical for calico-node during upgrades, as workload changes occur frequently. Consequently, there is a possibility of temporary desynchronization in the Calico dataplane. This can occasionally result in throttling when applying workload changes to the Calico dataplane.

Components versions¶

The following table lists the major components and their versions delivered in the Container Cloud 2.25.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

**Container Cloud release components versions**¶
Component	Application/Service	Version
Bare metal ^Updated	ambasador	1.38.17
	baremetal-dnsmasq	base-alpine-20231013162346
	baremetal-operator	base-alpine-20231101201729
	baremetal-provider	1.38.17
	bm-collective	base-alpine-20230929115341
	cluster-api-provider-baremetal	1.38.17
	ironic	yoga-jammy-20230914091512
	ironic-inspector	yoga-jammy-20230914091512
	ironic-prometheus-exporter	0.1-20230912104602
	kaas-ipam	base-alpine-20230911165405
	kubernetes-entrypoint	1.0.1-27d64fb-20230421151539
	mariadb	10.6.14-focal-20230912121635
	metallb-controller	0.13.9-0d8e8043-amd64
	metallb-speaker	0.13.9-0d8e8043-amd64
	syslog-ng	base-apline-20230914091214
IAM	iam ^Updated	2.5.8
	iam-controller ^Updated	1.38.17
	keycloak	21.1.1
Container Cloud	admission-controller ^Updated	1.38.17
	agent-controller ^Updated	1.38.17
	ceph-kcc-controller ^Updated	1.38.17
	cert-manager-controller	1.11.0-2
	cinder-csi-plugin ^New	1.27.2-8
	client-certificate-controller ^Updated	1.38.17
	configuration-collector ^New	1.38.17
	csi-attacher ^New	4.2.0-2
	csi-node-driver-registrar ^New	2.7.0-2
	csi-provisioner ^New	3.4.1-2
	csi-resizer ^New	1.7.0-2
	csi-snapshotter ^New	6.2.1-mcc-1
	event-controller ^Updated	1.38.17
	frontend ^Updated	1.38.17
	golang	1.20.4-alpine3.17
	iam-controller ^Updated	1.38.17
	kaas-exporter ^Updated	1.38.17
	kproxy ^Updated	1.38.17
	lcm-controller ^Updated	1.38.17
	license-controller ^Updated	1.38.17
	livenessprobe ^New	2.9.0-2
	machinepool-controller ^Updated	1.38.17
	mcc-haproxy	0.23.0-73-g01aa9b3
	metrics-server	0.6.3-2
	nginx ^Updated	1.38.17
	portforward-controller ^Updated	1.38.17
	proxy-controller ^Updated	1.38.17
	rbac-controller ^Updated	1.38.17
	registry	2.8.1-5
	release-controller ^Updated	1.38.17
	rhellicense-controller ^Updated	1.38.17
	scope-controller ^Updated	1.38.17
	storage-discovery ^Updated	1.38.17
	user-controller ^Updated	1.38.17
OpenStack ^Updated	openstack-cloud-controller-manager	1.27.2-8
	openstack-cluster-api-controller	1.38.17
	openstack-provider	1.38.17
	os-credentials-controller	1.38.17
VMware vSphere	mcc-keepalived ^Updated	0.23.0-73-g01aa9b3
	squid-proxy	0.0.1-10-g24a0d69
	vsphere-cluster-api-controller ^Updated	1.38.17
	vsphere-credentials-controller ^Updated	1.38.17
	vsphere-provider ^Updated	1.38.17
	vsphere-vm-template-controller ^Updated	1.38.17

Artifacts¶

This section lists the artifacts of components included in the Container Cloud release 2.25.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries ^Updated	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20231012141354
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20231012141354
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-113-4f8b843.tgz
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.38.17.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.38.17.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.38.17.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.38.17.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.38.17.tgz
	local-volume-provisioner	https://binary.mirantis.com/core/helm/local-volume-provisioner-1.38.17.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.38.17.tgz
Docker images ^Updated	ambasador	mirantis.azurecr.io/core/external/nginx:1.38.17
	baremetal-dnsmasq	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-alpine-20231013162346
	baremetal-operator	mirantis.azurecr.io/bm/baremetal-operator:base-alpine-20231101201729
	bm-collective	mirantis.azurecr.io/bm/bm-collective:base-alpine-20230929115341
	cluster-api-provider-baremetal	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.38.17
	ironic	mirantis.azurecr.io/openstack/ironic:yoga-jammy-20230914091512
	ironic-inspector	mirantis.azurecr.io/openstack/ironic-inspector:yoga-jammy-20230914091512
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20230912104602
	kaas-ipam	mirantis.azurecr.io/bm/kaas-ipam:base-alpine-20230911165405
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-27d64fb-20230421151539
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.14-focal-20230912121635
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.23.0-73-g01aa9b3
	metallb-controller	mirantis.azurecr.io/bm/metallb/controller:v0.13.9-0d8e8043-amd64
	metallb-speaker	mirantis.azurecr.io/bm/metallb/speaker:v0.13.9-0d8e8043-amd64
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-apline-20230914091214

Core artifacts¶

Artifact	Component	Paths
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.38.17.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.38.17.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.38.17.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.38.17.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.38.17.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.38.17.tgz
	cinder-csi-plugin ^New	https://binary.mirantis.com/core/helm/cinder-csi-plugin-1.38.17.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.38.17.tgz
	configuration-collector ^New	https://binary.mirantis.com/core/helm/configuration-collector-1.38.17.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.38.17.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.38.17.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.38.17.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.38.17.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.38.17.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.38.17.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.38.17.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.38.17.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.38.17.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.38.17.tgz
	metrics-server	https://binary.mirantis.com/core/helm/metrics-server-1.38.17.tgz
	openstack-cloud-controller-manager ^New	https://binary.mirantis.com/core/helm/openstack-cloud-controller-manager-1.38.17.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.38.17.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.38.17.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.38.17.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.38.17.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.38.17.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.38.17.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.38.17.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.38.17.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.38.17.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.38.17.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.38.17.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.38.17.tgz
	vsphere-vm-template-controller	https://binary.mirantis.com/core/helm/vsphere-vm-template-controller-1.38.17.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.38.17
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.38.17
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.38.17
	cert-manager-controller ^Updated	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-2
	cinder-csi-plugin ^New	mirantis.azurecr.io/lcm/kubernetes/cinder-csi-plugin:v1.27.2-8
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.38.17
	configuration-collector ^New	mirantis.azurecr.io/core/configuration-collector:1.38.17
	csi-attacher ^New	mirantis.azurecr.io/lcm/k8scsi/csi-attacher:v4.2.0-2
	csi-node-driver-registrar ^New	mirantis.azurecr.io/lcm/k8scsi/csi-node-driver-registrar:v2.7.0-2
	csi-provisioner ^New	mirantis.azurecr.io/lcm/k8scsi/csi-provisioner:v3.4.1-2
	csi-resizer ^New	mirantis.azurecr.io/lcm/k8scsi/csi-resizer:v1.7.0-2
	csi-snapshotter ^New	mirantis.azurecr.io/lcm/k8scsi/csi-snapshotter:v6.2.1-mcc-1
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.38.17
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.38.17
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.38.17
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.38.17
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.38.17
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.38.17
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.38.17
	livenessprobe ^New	mirantis.azurecr.io/lcm/k8scsi/livenessprobe:v2.9.0-2
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.38.17
	mcc-haproxy ^Updated	mirantis.azurecr.io/lcm/mcc-haproxy:v0.23.0-73-g01aa9b3
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.23.0-73-g01aa9b3
	metrics-server	mirantis.azurecr.io/lcm/metrics-server-amd64:v0.6.3-2
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.38.17
	openstack-cloud-controller-manager ^Updated	mirantis.azurecr.io/lcm/kubernetes/openstack-cloud-controller-manager:v1.27.2-8
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.38.17
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.38.17
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.38.17
	proxy-controller ^Updated	mirantis.azurecr.io/core/proxy-controller:1.38.17
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.38.17
	registry ^Updated	mirantis.azurecr.io/lcm/registry:v2.8.1-6
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.38.17
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.38.17
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.38.17
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-10-g24a0d69
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.38.17
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.38.17
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.38.17
	vsphere-vm-template-controller ^Updated	mirantis.azurecr.io/core/vsphere-vm-template-controller:1.38.17

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts ^Updated	iam	https://binary.mirantis.com/iam/helm/iam-2.5.8.tgz
Docker images	keycloak	mirantis.azurecr.io/iam/keycloak:0.6.0
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-27d64fb-20230421151539
	mariadb ^Updated	mirantis.azurecr.io/general/mariadb:10.6.14-focal-20230730124341

Security notes¶

The table below includes the total numbers of addressed unique and common CVEs by product component since the 2.24.5 patch release. The common CVEs are issues addressed across several images.

Addressed CVEs - summary¶
Container Cloud component	CVE type	Critical	High	Total
Kaas core	Unique	7	39	46
Kaas core	Common	54	305	359
Ceph	Unique	0	1	1
Ceph	Common	0	1	1
StackLight	Unique	0	5	5
StackLight	Common	0	13	13

Mirantis Security Portal

For the detailed list of fixed and existing CVEs across the Mirantis Container Cloud and MOSK products, refer to Mirantis Security Portal.

MOSK CVEs

For the number of fixed CVEs in the MOSK-related components including OpenStack and Tungsten Fabric, refer to MOSK 23.3: Security notes.

Update notes¶

This section describes the specific actions you as a cloud operator need to complete before or after your Container Cloud cluster update to the Cluster releases 17.0.0, 16.0.0, or 14.1.0.

Consider this information as a supplement to the generic update procedures published in Operations Guide: Automatic upgrade of a management cluster and Update a managed cluster.

Pre-update actions¶

Upgrade to Ubuntu 20.04 on baremetal-based clusters¶

The Cluster release series 14.x and 15.x are the last ones where Ubuntu 18.04 is supported on existing clusters. A Cluster release update to 17.0.0 or 16.0.0 is impossible for a cluster running on Ubuntu 18.04.

Therefore, if your cluster update is blocked, make sure that the operating system on all cluster nodes is upgraded to Ubuntu 20.04 as described in Operations Guide: Upgrade an operating system distribution.

Configure managed clusters with the etcd storage quota set¶

If your cluster has custom etcd storage quota set as described in Increase storage quota for etcd, before the management cluster upgrade to 2.25.0, configure LCMMachine resources:

Manually set the ucp_etcd_storage_quota parameter in LCMMachine resources of the cluster controller nodes:
```
spec:
 stateItemsOverwrites:
 deploy:
 ucp_etcd_storage_quota: "<custom_etcd_storage_quota_value>"
```
If the stateItemsOverwrites.deploy section is already set, append ucp_etcd_storage_quota to the existing parameters.

To obtain the list of the cluster LCMMachine resources:
```
kubectl -n <cluster_namespace> get lcmmachine
```
To patch the cluster LCMMachine resources of the Type control:
```
kubectl -n <cluster_namespace> edit lcmmachine <control_lcmmachine_name>
```
After the management cluster is upgraded to 2.25.0, update your managed cluster to the Cluster release 17.0.0 or 16.0.0.
Manually remove the ucp_etcd_storage_quota parameter from the stateItemsOverwrites.deploy section.

Allow the TCP port 12392 for management cluster nodes¶

The Cluster release 16.x and 17.x series are shipped with MKE 3.7.x. To ensure cluster operability after the update, verify that the TCP port 12392 is allowed in your network for the Container Cloud management cluster nodes.

For the full list of the required ports for MKE, refer to MKE Documentation: Open ports to incoming traffic.

Post-update actions¶

Migrate Ceph cluster to address storage devices using by-id¶

Container Cloud uses the device by-id identifier as the default method of addressing the underlying devices of Ceph OSDs. This is the only persistent device identifier for a Ceph cluster that remains stable after cluster upgrade or any other cluster maintenance.

Therefore, if your existing Ceph clusters are still utilizing the device names or device by-path symlinks, migrate them to the by-id format as described in Migrate Ceph cluster to address storage devices using by-id.

Point DHCP relays on routers to the new dhcp-lb IP address¶

If your managed cluster has multiple L2 segments using DHCP relays on the border switches, after the related management cluster automatically upgrades to Container Cloud 2.25.0, manually point the DHCP relays on your network infrastructure to the new IP address of the dhcp-lb service of the Container Cloud managed cluster in order to successfully provision new nodes or reprovision existing ones.

To obtain the new IP address:

kubectl -n kaas get service dhcp-lb

This change is required after the product has included the resolution of the issue related to the incorrect handling of DHCP egress traffic. The fix involves reconfiguring the external traffic policy for the dhcp-lb Kubernetes Service. For details about the issue, refer to the Kubernetes upstream bug.

See also

2.24.5¶

The Container Cloud patch release 2.24.5, which is based on the 2.24.2 major release, provides the following updates:

Support for the patch Cluster releases 14.0.4 and 15.0.4 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 23.2.3.
Security fixes for CVEs of Critical and High severity

This patch release also supports the latest major Cluster releases 14.0.1 and 15.0.1. And it does not support greenfield deployments based on deprecated Cluster releases 15.0.3, 15.0.2, 14.0.3, 14.0.2 along with 12.7.x and 11.7.x series. Use the latest available Cluster releases for new deployments instead.

For main deliverables of the parent Container Cloud releases of 2.24.5, refer to 2.24.0 and 2.24.1.

Artifacts¶

This section lists the components artifacts of the Container Cloud patch release 2.24.5. For artifacts of the Cluster releases introduced in 2.24.5, see patch Cluster releases 15.0.4 and 14.0.4.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Bare metal artifacts¶
Artifact	Component	Path
Binaries	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20230606121129
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20230606121129
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-104-6e2e82c.tgz
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.37.25.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.37.25.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.37.25.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.37.25.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.37.25.tgz
	local-volume-provisioner	https://binary.mirantis.com/core/helm/local-volume-provisioner-1.37.25.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.37.25.tgz
Docker images	ambasador ^Updated	mirantis.azurecr.io/core/external/nginx:1.37.25
	baremetal-dnsmasq	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-alpine-20230810152159
	baremetal-operator	mirantis.azurecr.io/bm/baremetal-operator:base-alpine-20230803175048
	bm-collective	mirantis.azurecr.io/bm/bm-collective:base-alpine-20230829084517
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.37.25
	ironic	mirantis.azurecr.io/openstack/ironic:yoga-focal-20230810113432
	ironic-inspector	mirantis.azurecr.io/openstack/ironic-inspector:yoga-focal-20230810113432
	ironic-prometheus-exporter ^Updated	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20230912104602
	kaas-ipam	mirantis.azurecr.io/bm/kaas-ipam:base-alpine-20230810155639
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-5359171-20230810125608
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.14-focal-20230730124341
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.22.0-75-g08569a8
	metallb-controller	mirantis.azurecr.io/bm/metallb/controller:v0.13.9-53df4a9c-amd64
	metallb-speaker	mirantis.azurecr.io/bm/metallb/speaker:v0.13.9-53df4a9c-amd64
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-apline-20230814110635

Core artifacts¶

Artifact	Component	Path
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.37.25.tgz
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.37.25.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.37.25.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.37.25.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.37.25.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.37.25.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.37.25.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.37.25.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.37.25.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.37.25.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.37.25.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.37.25.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.37.25.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.37.25.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.37.25.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.37.25.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.37.25.tgz
	metrics-server	https://binary.mirantis.com/core/helm/metrics-server-1.37.25.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.37.25.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.37.25.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.37.25.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.37.25.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.37.25.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.37.25.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.37.25.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.37.25.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.37.25.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.37.25.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.37.25.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.37.25.tgz
	vsphere-vm-template-controller	https://binary.mirantis.com/core/helm/vsphere-vm-template-controller-1.37.25.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.37.25
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.37.25
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.37.25
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-2
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.37.25
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.37.25
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.37.25
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.37.25
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.37.25
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.37.25
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.37.25
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.37.25
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.37.25
	mcc-haproxy	mirantis.azurecr.io/lcm/mcc-haproxy:v0.22.0-75-g08569a8
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.22.0-75-g08569a8
	metrics-server	mirantis.azurecr.io/lcm/metrics-server-amd64:v0.6.3-2
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.37.25
	openstack-cloud-controller-manager	mirantis.azurecr.io/lcm/kubernetes/openstack-cloud-controller-manager-amd64:v1.24.5-13
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.37.25
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.37.25
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.37.25
	proxy-controller ^Updated	mirantis.azurecr.io/core/proxy-controller:1.37.25
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.37.25
	registry ^Updated	mirantis.azurecr.io/lcm/registry:v2.8.1-5
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.37.25
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.37.25
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.37.25
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-10-g24a0d69
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.37.25
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.37.25
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.37.25
	vsphere-vm-template-controller ^Updated	mirantis.azurecr.io/core/vsphere-vm-template-controller:1.37.25

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts	iam	https://binary.mirantis.com/iam/helm/iam-2.5.4.tgz
Docker images	keycloak	mirantis.azurecr.io/iam/keycloak:0.6.0
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-27d64fb-20230421151539
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.12-focal-20230423170220

Security notes¶

In total, since Container Cloud 2.24.4, in 2.24.5, 21 Common Vulnerabilities and Exposures (CVE) have been fixed: 18 of critical and 3 of high severity.

The summary table contains the total number of unique CVEs along with the total number of issues fixed across the images.

The full list of the CVEs present in the current Container Cloud release is available at the Mirantis Security Portal.

Addressed CVEs - summary¶
Severity	Critical	High	Total
Unique CVEs	1	1	2
Total issues across images	18	3	21

Addressed CVEs - detailed¶
Image	Component name	CVE
core/external/nginx	libwebp	CVE-2023-4863 (High)
core/frontend	libwebp	CVE-2023-4863 (High)
lcm/kubernetes/openstack-cloud-controller-manager-amd64	busybox	CVE-2022-48174 (Critical)
	busybox-binsh	CVE-2022-48174 (Critical)
	ssl_client	CVE-2022-48174 (Critical)
lcm/registry	busybox	CVE-2022-48174 (Critical)
	busybox-binsh	CVE-2022-48174 (Critical)
	ssl_client	CVE-2022-48174 (Critical)
scale/curl-jq	busybox	CVE-2022-48174 (Critical)
	busybox-binsh	CVE-2022-48174 (Critical)
	ssl_client	CVE-2022-48174 (Critical)
stacklight/alertmanager-webhook-servicenow	busybox	CVE-2022-48174 (Critical)
	busybox-binsh	CVE-2022-48174 (Critical)
	ssl_client	CVE-2022-48174 (Critical)
stacklight/grafana-image-renderer	libwebp	CVE-2023-4863 (High)
stacklight/ironic-prometheus-exporter	busybox	CVE-2022-48174 (Critical)
	busybox-binsh	CVE-2022-48174 (Critical)
	ssl_client	CVE-2022-48174 (Critical)
stacklight/sf-reporter	busybox	CVE-2022-48174 (Critical)
	busybox-binsh	CVE-2022-48174 (Critical)
	ssl_client	CVE-2022-48174 (Critical)

2.24.4¶

The Container Cloud patch release 2.24.4, which is based on the 2.24.2 major release, provides the following updates:

Support for the patch Cluster releases 14.0.3 and 15.0.3 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release 23.2.2.
Support for the multi-rack topology on bare metal managed clusters
Support for configuration of the etcd storage quota
Security fixes for CVEs of Critical and High severity

This patch release also supports the latest major Cluster releases 14.0.1 and 15.0.1. And it does not support greenfield deployments based on deprecated Cluster releases 15.0.2, 14.0.2, along with 12.7.x and 11.7.x series. Use the latest available Cluster releases for new deployments instead.

For main deliverables of the parent Container Cloud releases of 2.24.4, refer to 2.24.0 and 2.24.1.

Enhancements¶

This section outlines new features and enhancements introduced in the Container Cloud patch release 2.24.4.

Configuration of the etcd storage quota¶

Added the capability to configure storage quota, which is 2 GB by default. You may need to increase the default etcd storage quota if etcd runs out of space and there is no other way to clean up the storage on your management or managed cluster.

Learn more

Operations Guide: Increase storage quota for etcd

Multi-rack topology for bare metal managed clusters¶

TechPreview

Added support for the multi-rack topology on bare metal managed clusters. Implementation of the multi-rack topology implies the use of Rack and MultiRackCluster objects that support configuration of BGP announcement of the cluster API load balancer address.

You can now create a managed cluster where cluster nodes including Kubernetes masters are distributed across multiple racks without L2 layer extension between them, and use BGP for announcement of the cluster API load balancer address and external addresses of Kubernetes load-balanced services.

Learn more

Artifacts¶

This section lists the components artifacts of the Container Cloud patch release 2.24.4. For artifacts of the Cluster releases introduced in 2.24.4, see patch Cluster releases 15.0.3 and 14.0.3.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20230606121129
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20230606121129
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-104-6e2e82c.tgz
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.37.24.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.37.24.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.37.24.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.37.24.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.37.24.tgz
	local-volume-provisioner	https://binary.mirantis.com/core/helm/local-volume-provisioner-1.37.24.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.37.24.tgz
Docker images	ambasador ^Updated	mirantis.azurecr.io/core/external/nginx:1.37.24
	baremetal-dnsmasq	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-alpine-20230810152159
	baremetal-operator	mirantis.azurecr.io/bm/baremetal-operator:base-alpine-20230803175048
	bm-collective ^Updated	mirantis.azurecr.io/bm/bm-collective:base-alpine-20230829084517
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.37.24
	ironic	mirantis.azurecr.io/openstack/ironic:yoga-focal-20230810113432
	ironic-inspector	mirantis.azurecr.io/openstack/ironic-inspector:yoga-focal-20230810113432
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20230531081117
	kaas-ipam	mirantis.azurecr.io/bm/kaas-ipam:base-alpine-20230810155639
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-5359171-20230810125608
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.14-focal-20230730124341
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.22.0-66-ga855169
	metallb-controller	mirantis.azurecr.io/bm/metallb/controller:v0.13.9-53df4a9c-amd64
	metallb-speaker	mirantis.azurecr.io/bm/metallb/speaker:v0.13.9-53df4a9c-amd64
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-apline-20230814110635

Core artifacts¶

Artifact	Component	Paths
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com//core/binbootstrap-darwin-1.37.24.tgz
	bootstrap-linux	https://binary.mirantis.com//core/binbootstrap-linux-1.37.24.tgz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.37.24.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.37.24.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.37.24.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.37.24.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.37.24.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.37.24.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.37.24.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.37.24.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.37.24.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.37.24.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.37.24.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.37.24.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.37.24.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.37.24.tgz
	mcc-cache-warmup	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.37.24.tgz
	metrics-server	https://binary.mirantis.com/core/helm/metrics-server-1.37.24.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.37.24.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.37.24.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.37.24.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.37.24.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.37.24.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.37.24.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.37.24.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.37.24.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.37.24.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.37.24.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.37.24.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.37.24.tgz
	vsphere-vm-template-controller	https://binary.mirantis.com/core/helm/vsphere-vm-template-controller-1.37.24.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.37.24
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.37.24
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.37.24
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-2
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.37.24
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.37.24
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.37.24
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.37.24
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.37.24
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.37.24
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.37.24
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.37.24
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.37.24
	mcc-haproxy ^Updated	mirantis.azurecr.io/lcm/mcc-haproxy:v0.22.0-66-ga855169
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.22.0-66-ga855169
	metrics-server	mirantis.azurecr.io/lcm/metrics-server-amd64:v0.6.3-2
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.37.24
	openstack-cloud-controller-manager	mirantis.azurecr.io/lcm/kubernetes/openstack-cloud-controller-manager-amd64:v1.24.5-10-g93314b86
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.37.24
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.37.24
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.37.24
	proxy-controller ^Updated	mirantis.azurecr.io/core/proxy-controller:1.37.24
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.37.24
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-4
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.37.24
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.37.24
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.37.24
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-10-g24a0d69
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.37.24
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.37.24
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.37.24
	vsphere-vm-template-controller ^Updated	mirantis.azurecr.io/core/vsphere-vm-template-controller:1.37.24

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts	iam	https://binary.mirantis.com/iam/helm/iam-2.5.4.tgz
Docker images	keycloak	mirantis.azurecr.io/iam/keycloak:0.6.0
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-27d64fb-20230421151539
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.12-focal-20230423170220

Security notes¶

In total, since Container Cloud 2.24.3, in 2.24.4, 18 Common Vulnerabilities and Exposures (CVE) have been fixed: 3 of critical and 15 of high severity.

The summary table contains the total number of unique CVEs along with the total number of issues fixed across the images.

The full list of the CVEs present in the current Container Cloud release is available at the Mirantis Security Portal.

Addressed CVEs - summary¶
Severity	Critical	High	Total
Unique CVEs	1	10	11
Total issues across images	3	15	18

Addressed CVEs - detailed¶
Image	Component name	CVE
iam/keycloak-gatekeeper	golang.org/x/crypto	CVE-2021-43565 (High)
		CVE-2022-27191 (High)
		CVE-2020-29652 (High)
	golang.org/x/net	CVE-2022-27664 (High)
		CVE-2021-33194 (High)
	golang.org/x/text	CVE-2021-38561 (High)
		CVE-2022-32149 (High)
	github.com/prometheus/client_golang	CVE-2022-21698 (High)
scale/psql-client	busybox	CVE-2022-48174 (Critical)
	busybox-binsh	CVE-2022-48174 (Critical)
	ssl_client	CVE-2022-48174 (Critical)
	libpq	CVE-2023-39417 (High)
	postgresql13-client	CVE-2023-39417 (High)
stacklight/alerta-web	grpcio	CVE-2023-33953 (High)
	libpq	CVE-2023-39417 (High)
	postgresql15-client	CVE-2023-39417 (High)
stacklight/pgbouncer	libpq	CVE-2023-39417 (High)
	postgresql-client	CVE-2023-39417 (High)

Addressed issues¶

The following issues have been addressed in the Container Cloud patch release 2.24.4 along with the patch Cluster releases 14.0.3 and 15.0.3.

[34200][Ceph] Fixed the watch command missing in the rook-ceph-tools Pod.
[34836][Ceph] Fixed ceph-disk-daemon spawning a lot of zombie processes.

2.24.3¶

The Container Cloud patch release 2.24.3, which is based on the 2.24.2 major release, provides the following updates:

Support for MKE 3.6.6. For details, see MKE documentation: MKE release notes.
Support for enablement of Kubernetes auditing and profiling options using the Container Cloud Cluster object on managed clusters. For details, see Configure Kubernetes auditing and profiling.
Updated docker-ee-cli to 20.10.18 for MCR 20.10.17 to fix the following CVEs: CVE-2023-28840, CVE-2023-28642, CVE-2022-41723.
Security fixes for CVEs of High severity.
Support for the patch Cluster releases 14.0.2 and 15.0.2 that represents Mirantis OpenStack for Kubernetes (MOSK) patch release. 23.2.1.

This patch release also supports the latest major Cluster releases 14.0.1 and 15.0.1. And it does not support greenfield deployments based on deprecated Cluster release 14.0.0 along with 12.7.x and 11.7.x series. Use the latest available Cluster releases instead.

For main deliverables of the parent Container Cloud releases of 2.24.3, refer to 2.24.0 and 2.24.1.

Artifacts¶

This section lists the components artifacts of the Container Cloud patch release 2.24.3. For artifacts of the Cluster releases introduced in 2.24.3, see Cluster releases 15.0.2 and 14.0.2.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20230606121129
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20230606121129
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-104-6e2e82c.tgz
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.37.23.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.37.23.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.37.23.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.37.23.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.37.23.tgz
	local-volume-provisioner	https://binary.mirantis.com/core/helm/local-volume-provisioner-1.37.23.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.37.23.tgz
Docker images	ambasador ^Updated	mirantis.azurecr.io/core/external/nginx:1.37.23
	baremetal-dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-alpine-20230810152159
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-alpine-20230803175048
	bm-collective ^Updated	mirantis.azurecr.io/bm/bm-collective:base-alpine-20230810134945
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.37.23
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:yoga-focal-20230810113432
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:yoga-focal-20230810113432
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20230531081117
	kaas-ipam ^Updated	mirantis.azurecr.io/bm/kaas-ipam:base-alpine-20230810155639
	kubernetes-entrypoint ^Updated	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-5359171-20230810125608
	mariadb ^Updated	mirantis.azurecr.io/general/mariadb:10.6.14-focal-20230730124341
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.22.0-63-g8f4f248
	metallb-controller ^Updated	mirantis.azurecr.io/bm/metallb/controller:v0.13.9-53df4a9c-amd64
	metallb-speaker ^Updated	mirantis.azurecr.io/bm/metallb/speaker:v0.13.9-53df4a9c-amd64
	syslog-ng ^Updated	mirantis.azurecr.io/bm/syslog-ng:base-apline-20230814110635

Core artifacts¶

Artifact	Component	Paths
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com//core/binbootstrap-darwin-1.37.23.tgz
	bootstrap-linux	https://binary.mirantis.com//core/binbootstrap-linux-1.37.23.tgz
Helm charts	admission-controller ^Updated	https://binary.mirantis.com/core/helm/admission-controller-1.37.23.tgz
	agent-controller ^Updated	https://binary.mirantis.com/core/helm/agent-controller-1.37.23.tgz
	ceph-kcc-controller ^Updated	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.37.23.tgz
	cert-manager ^Updated	https://binary.mirantis.com/core/helm/cert-manager-1.37.23.tgz
	client-certificate-controller ^Updated	https://binary.mirantis.com/core/helm/client-certificate-controller-1.37.23.tgz
	event-controller ^Updated	https://binary.mirantis.com/core/helm/event-controller-1.37.23.tgz
	iam-controller ^Updated	https://binary.mirantis.com/core/helm/iam-controller-1.37.23.tgz
	kaas-exporter ^Updated	https://binary.mirantis.com/core/helm/kaas-exporter-1.37.23.tgz
	kaas-public-api ^Updated	https://binary.mirantis.com/core/helm/kaas-public-api-1.37.23.tgz
	kaas-ui ^Updated	https://binary.mirantis.com/core/helm/kaas-ui-1.37.23.tgz
	lcm-controller ^Updated	https://binary.mirantis.com/core/helm/lcm-controller-1.37.23.tgz
	license-controller ^Updated	https://binary.mirantis.com/core/helm/license-controller-1.37.23.tgz
	machinepool-controller ^Updated	https://binary.mirantis.com/core/helm/machinepool-controller-1.37.23.tgz
	mcc-cache ^Updated	https://binary.mirantis.com/core/helm/mcc-cache-1.37.23.tgz
	mcc-cache-warmup ^Updated	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.37.23.tgz
	metrics-server ^Updated	https://binary.mirantis.com/core/helm/metrics-server-1.37.23.tgz
	openstack-provider ^Updated	https://binary.mirantis.com/core/helm/openstack-provider-1.37.23.tgz
	os-credentials-controller ^Updated	https://binary.mirantis.com/core/helm/os-credentials-controller-1.37.23.tgz
	portforward-controller ^Updated	https://binary.mirantis.com/core/helm/portforward-controller-1.37.23.tgz
	proxy-controller ^Updated	https://binary.mirantis.com/core/helm/proxy-controller-1.37.23.tgz
	rbac-controller ^Updated	https://binary.mirantis.com/core/helm/rbac-controller-1.37.23.tgz
	release-controller ^Updated	https://binary.mirantis.com/core/helm/release-controller-1.37.23.tgz
	rhellicense-controller ^Updated	https://binary.mirantis.com/core/helm/rhellicense-controller-1.37.23.tgz
	scope-controller ^Updated	https://binary.mirantis.com/core/helm/scope-controller-1.37.23.tgz
	squid-proxy ^Updated	https://binary.mirantis.com/core/helm/squid-proxy-1.37.23.tgz
	user-controller ^Updated	https://binary.mirantis.com/core/helm/user-controller-1.37.23.tgz
	vsphere-credentials-controller ^Updated	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.37.23.tgz
	vsphere-provider ^Updated	https://binary.mirantis.com/core/helm/vsphere-provider-1.37.23.tgz
	vsphere-vm-template-controller ^Updated	https://binary.mirantis.com/core/helm/vsphere-vm-template-controller-1.37.23.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.37.23
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.37.23
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.37.23
	cert-manager-controller ^Updated	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0-2
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.37.23
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.37.23
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.37.23
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.37.23
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.37.23
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.37.23
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.37.23
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.37.23
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.37.23
	mcc-haproxy ^Updated	mirantis.azurecr.io/lcm/mcc-haproxy:v0.22.0-63-g8f4f248
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.22.0-63-g8f4f248
	metrics-server	mirantis.azurecr.io/lcm/metrics-server-amd64:v0.6.3-2
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.37.23
	openstack-cloud-controller-manager	mirantis.azurecr.io/lcm/kubernetes/openstack-cloud-controller-manager-amd64:v1.24.5-10-g93314b86
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.37.23
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.37.23
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.37.23
	proxy-controller ^Updated	mirantis.azurecr.io/core/proxy-controller:1.37.23
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.37.23
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-4
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.37.23
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.37.23
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.37.23
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-10-g24a0d69
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.37.23
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.37.23
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.37.23
	vsphere-vm-template-controller ^Updated	mirantis.azurecr.io/core/vsphere-vm-template-controller:1.37.23

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts	iam ^Updated	https://binary.mirantis.com/iam/helm/iam-2.5.4.tgz
Docker images	keycloak	mirantis.azurecr.io/iam/keycloak:0.6.0
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-27d64fb-20230421151539
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.12-focal-20230423170220

Security notes¶

In total, since Container Cloud 2.24.1, in 2.24.3, 63 Common Vulnerabilities and Exposures (CVE) with high severity have been fixed.

The summary table contains the total number of unique CVEs along with the total number of issues fixed across the images.

The full list of the CVEs present in the current Container Cloud release is available at the Mirantis Security Portal.

Addressed CVEs - summary¶
Severity	Critical	High	Total
Unique CVEs	0	15	15
Total issues across images	0	63	63

Addressed CVEs - detailed¶
Image	Component name	CVE
bm/external/metallb/controller	libcrypto3	CVE-2023-0464 (High)
		CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
		CVE-2023-0464 (High)
	golang.org/x/net	CVE-2022-41723 (High)
bm/external/metallb/speaker	libcrypto3	CVE-2023-2650 (High)
		CVE-2023-0464 (High)
	libssl3	CVE-2023-0464 (High)
		CVE-2023-2650 (High)
	golang.org/x/net	CVE-2022-41723 (High)
core/external/cert-manager-cainjector	golang.org/x/net	CVE-2022-41723 (High)
core/external/cert-manager-controller	golang.org/x/net	CVE-2022-41723 (High)
core/external/cert-manager-webhook	golang.org/x/net	CVE-2022-41723 (High)
core/external/nginx	nghttp2-libs	CVE-2023-35945 (High)
core/frontend	nghttp2-libs	CVE-2023-35945 (High)
lcm/external/csi-attacher	github.com/prometheus/client_golang	CVE-2022-21698 (High)
	golang.org/x/net	CVE-2022-27664 (High)
	golang.org/x/text	CVE-2022-32149 (High)
	gopkg.in/yaml.v3	CVE-2022-28948 (High)
lcm/external/csi-node-driver-registrar	github.com/prometheus/client_golang	CVE-2022-21698 (High)
	golang.org/x/net	CVE-2022-27664 (High)
	golang.org/x/text	CVE-2022-32149 (High)
lcm/external/csi-provisioner	golang.org/x/crypto	CVE-2021-43565 (High)
		CVE-2022-27191 (High)
	github.com/prometheus/client_golang	CVE-2022-21698 (High)
	golang.org/x/net	CVE-2022-27664 (High)
	golang.org/x/text	CVE-2022-32149 (High)
	gopkg.in/yaml.v3	CVE-2022-28948 (High)
lcm/external/csi-resizer	github.com/prometheus/client_golang	CVE-2022-21698 (High)
	golang.org/x/net	CVE-2022-27664 (High)
	golang.org/x/text	CVE-2022-32149 (High)
	gopkg.in/yaml.v3	CVE-2022-28948 (High)
lcm/external/csi-snapshotter	github.com/prometheus/client_golang	CVE-2022-21698 (High)
	golang.org/x/net	CVE-2022-27664 (High)
	golang.org/x/text	CVE-2022-32149 (High)
	gopkg.in/yaml.v3	CVE-2022-28948 (High)
lcm/external/livenessprobe	golang.org/x/text	CVE-2021-38561 (High)
		CVE-2022-32149 (High)
	github.com/prometheus/client_golang	CVE-2022-21698 (High)
	golang.org/x/net	CVE-2022-27664 (High)
lcm/kubernetes/cinder-csi-plugin-amd64	libpython3.7-minimal	CVE-2021-3737 (High)
		CVE-2020-10735 (High)
		CVE-2022-45061 (High)
		CVE-2015-20107 (High)
	libpython3.7-stdlib	CVE-2021-3737 (High)
		CVE-2020-10735 (High)
		CVE-2022-45061 (High)
		CVE-2015-20107 (High)
	python3.7	CVE-2021-3737 (High)
		CVE-2020-10735 (High)
		CVE-2022-45061 (High)
		CVE-2015-20107 (High)
	python3.7-minimal	CVE-2021-3737 (High)
		CVE-2020-10735 (High)
		CVE-2022-45061 (High)
		CVE-2015-20107 (High)
	libssl1.1	CVE-2023-2650 (High)
		CVE-2023-0464 (High)
	openssl	CVE-2023-2650 (High)
		CVE-2023-0464 (High)
lcm/mcc-haproxy	nghttp2-libs	CVE-2023-35945 (High)
openstack/ironic	cryptography	CVE-2023-2650 (High)
openstack/ironic-inspector	cryptography	CVE-2023-2650 (High)

Addressed issues¶

The following issues have been addressed in the Container Cloud patch release 2.24.3 along with the patch Cluster releases 14.0.2 and 15.0.2.

[34638][BM] Fixed the issue with failure to delete a management cluster due to the issue with secrets during machine deletion.
[34220][BM] Fixed the issue with ownerReferences being lost for HardwareData after pivoting during a management cluster bootstrap.
[34280][LCM] Fixed the issue with no cluster reconciles generated if a cluster is stuck on waiting for agents upgrade.
[33439][TLS] Fixed the issue with client-certificate-controller silently replacing user-provided key if PEM header and key format do not match.
[33686][audit] Fixed the issue with rules provided by the docker auditd preset not covering the Sysdig Docker CIS benchmark.
[34080][StackLight] Fixed the issue with missing events in OpenSearch that have lastTimestamp set to null and eventTime set to a non-null value.

2.24.2¶

The Container Cloud major release 2.24.2 based on 2.24.0 and 2.24.1 provides the following:

Introduces support for the major Cluster release 15.0.1 that is based on the Cluster release 14.0.1 and represents Mirantis OpenStack for Kubernetes (MOSK) 23.2. This Cluster release is based on the updated version of Mirantis Kubernetes Engine 3.6.5 with Kubernetes 1.24 and Mirantis Container Runtime 20.10.17.
Supports the latest Cluster release 14.0.1.
Does not support greenfield deployments based on deprecated Cluster release 14.0.0 along with 12.7.x and 11.7.x series. Use the latest available Cluster releases of the series instead.

For main deliverables of the Container Cloud release 2.24.2, refer to its parent release 2.24.0:

Caution

Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

2.24.1¶

The Container Cloud patch release 2.24.1 based on 2.24.0 includes updated baremetal-operator, admission-controller, and iam artifacts and provides hot fixes for the following issues:

[34218] Fixed the issue with the iam-keycloak Pod being stuck in the Pending state during Keycloak upgrade to version 21.1.1.
[34247] Fixed the issue with MKE backup failing during cluster update due to wrong permissions in the etcd backup directory. If the issue still persists, which may occur on clusters that were originally deployed using early Container Cloud releases delivered in 2020-2021, follow the workaround steps described in Known issues: LCM.

Note

Container Cloud patch release 2.24.1 does not introduce new Cluster releases.

For main deliverables of the Container Cloud release 2.24.1, refer to its parent release 2.24.0:

Caution

2.24.0¶

Important

Container Cloud 2.24.0 has been successfully applied to a certain number of clusters. The 2.24.0 related documentation content fully applies to these clusters.

If your cluster started to update but was reverted to the previous product version or the update is stuck, you automatically receive the 2.24.1 patch release with the bug fixes to unblock the update to the 2.24 series.

There is no impact on the cluster workloads. For details on the patch release, see 2.24.1.

The Mirantis Container Cloud GA release 2.24.0:

Introduces support for the Cluster release 14.0.0 that is based on Mirantis Container Runtime 20.10.17 and Mirantis Kubernetes Engine 3.6.5 with Kubernetes 1.24.
Supports the latest major and patch Cluster releases of the 12.7.x series that supports Mirantis OpenStack for Kubernetes (MOSK) 23.1 series.
Does not support greenfield deployments on deprecated Cluster releases 12.7.3, 11.7.4, or earlier patch releases, 12.5.0, or 11.7.0. Use the latest available Cluster releases of the series instead.

Caution

Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.24.0.

Enhancements¶

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.24.0. For the list of enhancements in the Cluster release 14.0.0 that is introduced by the Container Cloud release 2.24.0, see the 14.0.0.

Automated upgrade of operating system on bare metal clusters
Support for WireGuard on bare metal clusters
MetalLB configuration changes for bare metal and vSphere
Support for RHEL 8.7 on the vSphere provider
Custom flavors for Octavia on OpenStack-based clusters
Deletion of persistent volumes during an OpenStack-based cluster deletion
Support for Keycloak Quarkus
Custom host names for cluster machines
Parallel update of worker nodes
Cache warm-up for managed clusters
Support for auditd
Enhancements for TLS certificates configuration
Graceful cluster reboot using web UI
Creation and deletion of bare metal host credentials using web UI
Node labeling improvements in web UI
Documentation enhancements

Automated upgrade of operating system on bare metal clusters¶

Support status of the feature

Since MOSK 23.2, the feature is generally available for MOSK clusters.
Since Container Cloud 2.24.2, the feature is generally available for any type of bare metal clusters.
Since Container Cloud 2.24.0, the feature is available as Technology Preview for management and regional clusters only.

Implemented automatic in-place upgrade of an operating system (OS) distribution on bare metal clusters. The OS upgrade occurs as part of cluster update that requires machines reboot. The OS upgrade workflow is as follows:

The distribution ID value is taken from the id field of the distribution from the allowedDistributions list in the spec of the ClusterRelease object.
The distribution that has the default: true value is used during update. This distribution ID is set in the spec:providerSpec:value:distribution field of the Machine object during cluster update.

On management and regional clusters, the operating system upgrades automatically during cluster update. For managed clusters, an in-place OS distribution upgrade should be performed between cluster updates. This scenario implies a machine cordoning, draining, and reboot.

Warning

It is not mandatory to upgrade all machines at once. You can upgrade them one by one or in small batches, for example, if the maintenance window is limited in time.

Otherwise, the Cluster release update of the Ubuntu 20.04-based managed clusters will become impossible as of Container Cloud 2.29.0 with Ubuntu 22.04 as the only supported version.

Management cluster update to Container Cloud 2.29.1 will be blocked if at least one node of any related managed cluster is running Ubuntu 20.04.

Learn more

Support for WireGuard on bare metal clusters¶

TechPreview

Added initial Technology Preview support for WireGuard that enables traffic encryption on the Kubernetes workloads network. Set secureOverlay: true in the Cluster object during deployment of management, regional, or managed bare metal clusters to enable WireGuard encryption.

Also, added the possibility to configure the maximum transmission unit (MTU) size for Calico that is required for the WireGuard functionality and allows maximizing network performance.

Note

For MOSK-based deployments, the feature support is available since MOSK 23.2.

Learn more

MetalLB configuration changes for bare metal and vSphere¶

For management and regional clusters

Caution

For managed clusters, this object is available as Technology Preview and will become generally available in one of the following Container Cloud releases.

Introduced the following MetalLB configuration changes and objects related to address allocation and announcement of services LB for bare metal and vSphere providers:

Introduced the MetalLBConfigTemplate object for bare metal and the MetalLBConfig object for vSphere to be used as default and recommended.
For vSphere, during creation of clusters of any type, now a separate MetalLBConfig object is created instead of corresponding settings in the Cluster object.
The use of either Subnet objects without the new MetalLB objects or the configInline MetalLB value of the Cluster object is deprecated and will be removed in one of the following releases.
If the MetalLBConfig object is not used for MetalLB configuration related to address allocation and announcement of services LB, then automated migration applies during creation of clusters of any type or cluster update to Container Cloud 2.24.0.

During automated migration, the MetalLBConfig and MetalLBConfigTemplate objects for bare metal or the MetalLBConfig for vSphere are created and contents of the MetalLB chart configInline value is converted to the parameters of the MetalLBConfigTemplate object for bare metal or of the MetalLBConfig object for vSphere.

The following changes apply to the bare metal bootstrap procedure:

Moved the following environment variables from cluster.yaml.template to the dedicated ipam-objects.yaml.template:
- BOOTSTRAP_METALLB_ADDRESS_POOL
- KAAS_BM_BM_DHCP_RANGE
- SET_METALLB_ADDR_POOL
- SET_LB_HOST
Modified the default network configuration. Now it includes a bond interface and separated PXE and management networks. Mirantis recommends using separate PXE and management networks for management and regional clusters.

Learn more

Support for RHEL 8.7 on the vSphere provider¶

TechPreview

Added support for RHEL 8.7 on the vSphere-based management, regional, and managed clusters.

Custom flavors for Octavia on OpenStack-based clusters¶

Implemented the possibility to use custom Octavia Amphora flavors that you can enable in spec:providerSpec section of the Cluster object using serviceAnnotations:loadbalancer.openstack.org/flavor-id during management or regional cluster deployment.

Note

For managed clusters, you can enable the feature through the Container Cloud API. The web UI functionality will be added in one of the following Container Cloud releases.

Learn more

Deployment Guide: Bootstrap a management cluster

Deletion of persistent volumes during an OpenStack-based cluster deletion¶

Completed the development of persistent volumes deletion during an OpenStack-based managed cluster deletion by implementing the Delete all volumes in the cluster check box in the cluster deletion menu of the Container Cloud web UI.

Support for Keycloak Quarkus¶

Upgraded the Keycloak major version from 18.0.0 to 21.1.1. For the list of new features and enhancements, see Keycloak Release Notes.

The upgrade path is fully automated. No data migration or custom LCM changes are required.

Important

After the Keycloak upgrade, access the Keycloak Admin Console using the new URL format: https://<keycloak.ip>/auth instead of https://<keycloak.ip>. Otherwise, the Resource not found error displays in a browser.

Learn more

Custom host names for cluster machines¶

TechPreview

Added initial Technology Preview support for custom host names of machines on any supported provider and any cluster type. When enabled, any machine host name in a particular region matches the related Machine object name. For example, instead of the default kaas-node-<UID>, a machine host name will be master-0. The custom naming format is more convenient and easier to operate with.

You can enable the feature before or after management or regional cluster deployment. If enabled after deployment, custom host names will apply to all newly deployed machines in the region. Existing host names will remain the same.

Learn more

Parallel update of worker nodes¶

TechPreview

Added initial Technology Preview support for parallelizing of node update operations that significantly improves the efficiency of your cluster. To configure the parallel node update, use the following parameters located under spec.providerSpec of the Cluster object:

maxWorkerUpgradeCount - maximum number of worker nodes for simultaneous update to limit machine draining during update
maxWorkerPrepareCount - maximum number of workers for artifacts downloading to limit network load during update

Note

For MOSK clusters, you can start using this feature during cluster update from 23.1 to 23.2. For details, see MOSK documentation: Parallelizing node update operations.

Learn more

Operations Guide: Configure the parallel update of worker nodes

Cache warm-up for managed clusters¶

Implemented the CacheWarmupRequest resource to predownload, aka warm up, a list of artifacts included in a given set of Cluster releases into the mcc-cache service only once per release. The feature facilitates and speeds up deployment and update of managed clusters.

After a successful cache warm-up, the object of the CacheWarmupRequest resource is automatically deleted from the cluster and cache remains for managed clusters deployment or update until next Container Cloud auto-upgrade of the management or regional cluster.

Caution

If the disk space for cache runs out, the cache for the oldest object is evicted. To avoid running out of space in the cache, verify and adjust its size before each cache warm-up.

Note

For MOSK-based deployments, the feature support is available since MOSK 23.2.

Learn more

Support for auditd¶

TechPreview

Added initial Technology Preview support for the Linux Audit daemon auditd to monitor activity of cluster processes on any type of Container Cloud cluster. The feature is an essential requirement for many security guides that enables auditing of any cluster process to detect potential malicious activity.

You can enable and configure auditd either during or after cluster deployment using the Cluster object.

Note

For MOSK-based deployments, the feature support is available since MOSK 23.2.

Learn more

Enhancements for TLS certificates configuration¶

TechPreview

Enhanced TLS certificates configuration for cluster applications:

Added support for configuration of TLS certificates for MKE on management or regional clusters to the existing support on managed clusters.
Implemented the ability to configure TLS certificates using the Container Cloud web UI through the Security section located in the More > Configure cluster menu.

Learn more

Operations Guide: Configure TLS certificates for cluster applications

Graceful cluster reboot using web UI¶

Expanded the capability to perform a graceful reboot on a management, regional, or managed cluster for all supported providers by adding the Reboot machines option to the cluster menu in the Container Cloud web UI. The feature allows for a rolling reboot of all cluster machines without workloads interruption. The reboot occurs in the order of cluster upgrade policy.

Note

For MOSK-based deployments, the feature support is available since MOSK 23.2.

Learn more

Creation and deletion of bare metal host credentials using web UI¶

Improved management of bare metal host credentials using the Container Cloud web UI:

Added the Add Credential menu to the Credentials tab. The feature facilitates association of credentials with bare metal hosts created using the BM Hosts tab.
Implemented automatic deletion of credentials during deletion of bare metal hosts after deletion of managed cluster.

Learn more

Node labeling improvements in web UI¶

Improved the Node Labels menu in the Container Cloud web UI by making it more intuitive. Replaced the greyed out (disabled) label names with the No labels have been assigned to this machine. message and the Add a node label button link.

Also, added the possibility to configure node labels for machine pools after deployment using the More > Configure Pool option.

Learn more

Operations Guide: Add a bare metal machine using web UI

Documentation enhancements¶

On top of continuous improvements delivered to the existing Container Cloud guides, added the documentation on managing Ceph OSDs with a separate metadata device.

Learn more

Addressed issues¶

The following issues have been addressed in the Mirantis Container Cloud release 2.24.0 along with the Cluster release 14.0.0. For the list of hot fixes delivered in the 2.24.1 patch release, see 2.24.1.

[5981] Fixed the issue with upgrade of a cluster containing more than 120 nodes getting stuck on one node with errors about IP addresses exhaustion in the docker logs. On existing clusters, after updating to the Cluster release 14.0.0 or later, you can optionally remove the abandoned mke-overlay network using docker network rm mke-overlay.
[29604] Fixed the issue with the false positive failed to get kubeconfig error occurring on the Waiting for TLS settings to be applied stage during TLS configuration.
[29762] Fixed the issue with a wrong IP address being assigned after the MetalLB controller restart.
[30635] Fixed the issue with the pg_autoscaler module of Ceph Manager failing with the pool <poolNumber> has overlapping roots error if a Ceph cluster contains a mix of pools with deviceClass either explicitly specified or not specified.
[30857] Fixed the issue with irrelevant error message displaying in the osd-prepare Pod during the deployment of Ceph OSDs on removable devices on AMD nodes. Now, the error message clearly states that removable devices (with hotplug enabled) are not supported for deploying Ceph OSDs. This issue has been addressed since the Cluster release 14.0.0.
[30781] Fixed the issue with cAdvisor failing to collect metrics on CentOS-based deployments. Missing metrics affected the KubeContainersCPUThrottlingHigh alert and the following Grafana dashboards: Kubernetes Containers, Kubernetes Pods, and Kubernetes Namespaces.
[31288] Fixed the issue with Fluentd agent failing and the fluentd-logs Pods reporting the maximum open shards limit error, thus preventing OpenSearch to accept new logs. The fix enables the possibility to increase the limit for maximum open shards using cluster.max_shards_per_node. For details, see Tune StackLight for long-term log retention.
[31485] Fixed the issue with Elasticsearch Curator not deleting indices according to the configured retention period on any type of Container Cloud clusters.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud releases 2.24.0 and 2.24.1 including the Cluster release 14.0.0.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal
LCM
Ceph
Update

Bare metal¶

[42386] A load balancer service does not obtain the external IP address¶

Due to the MetalLB upstream issue, a load balancer service may not obtain the external IP address.

To work around the issue, make a dummy change to the service object where external IP is <pending>:

Identify the service that is stuck:

kubectl get svc -A | grep pending

Example of system response:

stacklight  iam-proxy-prometheus  LoadBalancer  10.233.28.196  <pending>  443:30430/TCP

Add an arbitrary label to the service that is stuck. For example:

kubectl label svc -n stacklight iam-proxy-prometheus reconcile=1

Example of system response:

service/iam-proxy-prometheus labeled

Verify that the external IP was allocated to the service:

kubectl get svc -n stacklight iam-proxy-prometheus

Example of system response:

NAME                  TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)        AGE
iam-proxy-prometheus  LoadBalancer  10.233.28.196  10.0.34.108  443:30430/TCP  12d

[36131] Changes in ‘IpamHost’ are not propagated to ‘LCMMachine’¶

Fixed in 17.0.0 and 16.0.0

During netplan configuration after cluster deployment, changes in the IpamHost object are not propagated to LCMMachine.

The workaround is to manually add any new label to the labels section of the Machine object for the target host, which triggers machine reconciliation and propagates network changes.

[35429] The WireGuard interface does not have the IPv4 address assigned¶

Fixed in 17.0.0 and 16.0.0

Due to the upstream Calico issue, on clusters with WireGuard enabled, the WireGuard interface on a node may not have the IPv4 address assigned. This leads to broken inter-Pod communication between the affected node and other cluster nodes.

The node is affected if the IP address is missing on the WireGuard interface:

ip a show wireguard.cali

Example of system response:

40: wireguard.cali: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue state UNKNOWN group default qlen 1000 link/none

The workaround is to manually restart the calico-node Pod to allocate the IPv4 address on the WireGuard interface:

docker restart $(docker ps -f "label=name=Calico node" -q)

[34280] No reconcile events generated during cluster update¶

Fixed in 15.0.2 and 14.0.2

The cluster update is stuck on waiting for agents to upgrade with the following message in the cluster status:

Helm charts are not installed(upgraded) yet. Not ready releases: managed-lcm-api

The workaround is to retrigger the cluster update, for example, by adding an annotation to the cluster object:

Log in to a local machine where your management cluster kubeconfig is located and where kubectl is installed.
Open the management Cluster object for editing:
```
kubectl edit cluster <mgmtClusterName>
```
Set the annotation force-reconcile: true.

[34210] Helm charts installation failure during cluster update¶

Fixed in 14.0.0

The cluster update is blocked with the following message in the cluster status:

Helm charts are not installed(upgraded) yet.
Not ready releases: iam, managed-lcm-api, admission-controller, baremetal-operator.

Workaround:

Log in to a local machine where your management cluster kubeconfig is located and where kubectl is installed.
Open the baremetal-operator deployment object for editing:
```
kubectl edit deploy -n kaas baremetal-operator
```
Modify the image that the init container and the container are using to mirantis.azurecr.io/bm/baremetal-operator:base-alpine-20230721153358.

The baremetal-operator pods will be re-created, and the cluster update will get unblocked.

[33936] Deletion failure of a controller node during machine replacement¶

Fixed in 17.0.1 and 16.0.1 for MKE 3.7.2

Resolving dependency Service dhcp-lb in namespace kaas failed: \
the server was unable to return a response in the time allotted,\
but may still be processing the request (get endpoints dhcp-lb).

As a workaround, delete the Pod that is stuck to retrigger the node deletion.

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

[20736] Region deletion failure after regional deployment failure¶

If a baremetal-based regional cluster deployment fails before pivoting is done, the corresponding region deletion fails.

Workaround:

Using the command below, manually delete all possible traces of the failed regional cluster deployment, including but not limited to the following objects that contain the kaas.mirantis.com/region label of the affected region:

cluster
machine
baremetalhost
baremetalhostprofile
l2template
subnet
ipamhost
ipaddr

kubectl delete <objectName> -l kaas.mirantis.com/region=<regionName>

Warning

Do not use the same region name again after the regional cluster deployment failure since some objects that reference the region name may still exist.

LCM¶

[31186,34132] Pods get stuck during MariaDB operations¶

During MariaDB operations on a management cluster, Pods may get stuck in continuous restarts with the following example error:

[ERROR] WSREP: Corrupt buffer header: \
addr: 0x7faec6f8e518, \
seqno: 3185219421952815104, \
size: 909455917, \
ctx: 0x557094f65038, \
flags: 11577. store: 49, \
type: 49

Workaround:

Create a backup of the /var/lib/mysql directory on the mariadb-server Pod.
Verify that other replicas are up and ready.
Remove the galera.cache file for the affected mariadb-server Pod.
Remove the affected mariadb-server Pod or wait until it is automatically restarted.

After Kubernetes restarts the Pod, the Pod clones the database in 1-2 minutes and restores the quorum.

[32761] Node cleanup fails due to remaining devices¶

Fixed in 17.1.0 and 16.1.0

The following symptoms mean that a cluster can be affected:

A node was configured to use thin pool LVs. For example, it had the OpenStack Cinder role in the past.
A bare metal node deployment flaps between provisioninig and deprovisioning states.
In the Ansible provisioner logs, the following example warnings are growing:
```
88621.log:7389:2023-06-22 16:30:45.109 88621 ERROR ansible.plugins.callback.ironic_log
[-] Ansible task clean : fail failed on node 14eb0dbc-c73a-4298-8912-4bb12340ff49:
{'msg': 'There are more devices to clean', '_ansible_no_log': None, 'changed': False}
```
Important

There are more devices to clean is a regular warning indicating some in-progress tasks. But if the number of such warnings is growing along with the node flapping between provisioninig and deprovisioning states, the cluster is highly likely affected by the issue.

As a workaround, erase disks manually using any preferred tool.

[34247] MKE backup fails during cluster update¶

Fixed in 14.0.0

MKE backup may fail during update of a management, regional, or managed cluster due to wrong permissions in the etcd backup /var/lib/docker/volumes/ucp-backup/_data directory.

The issue affects only clusters that were originally deployed using early Container Cloud releases delivered in 2020-2021.

Workaround:

Fix permissions on all affected nodes:

chown -R nobody:nogroup /var/lib/docker/volumes/ucp-backup/_data

Using the admin kubeconfig, increase the mkeUpgradeAttempts value:
1. Open the LCMCluster object of the management cluster for editing:
```
kubectl edit lcmcluster <mgmtClusterName>
```
2. In the mkeUpgradeAttempts field, increase the value to 6. Once done, MKE backup retriggers automatically.

[30294] Replacement of a master node is stuck on the calico-node Pod start¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a master node on a cluster of any type, the calico-node Pod fails to start on a new node that has the same IP address as the node being replaced.

Workaround:

From a CLI with an MKE client bundle, create a shell alias to start calicoctl using the mirantis/ucp-dsinfo image:

Since MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/key.pem \
-e ETCD_CA_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/ca.pem \
-e ETCD_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/cert.pem \
-v /var/run/calico:/var/run/calico \
-v /var/lib/docker/volumes/ucp-kv-certs/_data:/var/lib/docker/volumes/ucp-kv-certs/_data:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl \
"

Before MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
-e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
-e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
-v /var/run/calico:/var/run/calico \
-v ucp-node-certs:/ucp-node-certs:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl --allow-version-mismatch \
"

In the above command, replace the following values with the corresponding settings of the affected cluster:

<etcdEndpoint> is the etcd endpoint defined in the Calico configuration file. For example, ETCD_ENDPOINTS=127.0.0.1:12378
<mkeVersion> is the MKE version installed on your cluster. For example, mirantis/ucp-dsinfo:3.5.7.

Verify the node list on the cluster:
```
kubectl get node
```
Compare this list with the node list in Calico to identify the old node:
```
calicoctl get node -o wide
```

Remove the old node from Calico:

calicoctl delete node kaas-node-<nodeID>

[5782] Manager machine fails to be deployed during node replacement¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a manager machine, the following problems may occur:

The system adds the node to Docker swarm but not to Kubernetes
The node Deployment gets stuck with failed RethinkDB health checks

Workaround:

Delete the failed node.
Wait for the MKE cluster to become healthy. To monitor the cluster status:
1. Log in to the MKE web UI as described in Connect to the Mirantis Kubernetes Engine web UI.
2. Monitor the cluster status as described in MKE Operations Guide: Monitor an MKE cluster with the MKE web UI.
Deploy a new node.

[5568] The calico-kube-controllers Pod fails to clean up resources¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During the unsafe or forced deletion of a manager machine running the calico-kube-controllers Pod in the kube-system namespace, the following issues occur:

The calico-kube-controllers Pod fails to clean up resources associated with the deleted node
The calico-node Pod may fail to start up on a newly created node if the machine is provisioned with the same IP address as the deleted machine had

As a workaround, before deletion of the node running the calico-kube-controllers Pod, cordon and drain the node:

kubectl cordon <nodeName>
kubectl drain <nodeName>

Ceph¶

[34820] The Ceph ‘rook-operator’ fails to connect to RGW on FIPS nodes¶

Fixed in 17.1.0 and 16.1.0

Due to the upstream Ceph issue, on clusters with the Federal Information Processing Standard (FIPS) mode enabled, the Ceph rook-operator fails to connect to Ceph RADOS Gateway (RGW) pods.

As a workaround, do not place Ceph RGW pods on nodes where FIPS mode is enabled.

[34599] Ceph ‘ClusterWorkloadLock’ blocks upgrade from 2.23.5 to 2.24.1¶

On management clusters based on Ubuntu 18.04, after the cluster starts upgrading from 2.23.5 to 2.24.1, all controller machines are stuck in the In Progress state with the Distribution update in progress hover message displaying in the Container Cloud web UI.

The issue is caused by clusterworkloadlock containing the outdated release name in the status.release field, which blocks LCM Controller to proceed with machine upgrade. This behavior is caused by a complete removal of the ceph-controller chart from management clusters and a failed ceph-clusterworkloadlock removal.

The workaround is to manually remove ceph-clusterworkloadlock from the management cluster to unblock upgrade:

kubectl delete clusterworkloadlock ceph-clusterworkloadlock

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

Update¶

[33438] ‘CalicoDataplaneFailuresHigh’ alert is firing during cluster update¶

During cluster update of a managed bare metal cluster, the false positive CalicoDataplaneFailuresHigh alert may be firing. Disregard this alert, which will disappear once cluster update succeeds.

Components versions¶

The following table lists the major components and their versions delivered in the Container Cloud releases 2.24.0 - 2.24.2.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

**Container Cloud release components versions**¶
Component	Application/Service	Version
Bare metal ^Updated	ambassador	1.37.15
	baremetal-operator	base-alpine-20230607164516 ^2.24.0
		base-alpine-20230721153358 ^2.24.1(2)
	baremetal-public-api	1.37.15
	baremetal-provider	1.37.15
	ironic	yoga-focal-20230605060019
	kaas-ipam	base-alpine-20230614192933
	keepalived	0.22.0-49-g9618f2a
	local-volume-provisioner	2.5.0-4
	mariadb	10.6.12-focal-20230606052917
IAM ^Updated	iam	2.5.1 ^2.24.0
		2.5.3 ^2.24.1(2)
	iam-controller	1.37.15
	keycloak	21.1.1
Container Cloud ^Updated	admission-controller	1.37.15 ^2.24.0
		1.37.16 ^2.24.1
		1.37.19 ^2.24.2
	agent-controller	1.37.15
	byo-credentials-controller ^Removed	n/a
	byo-provider ^Removed	n/a
	ceph-kcc-controller	1.37.15
	cert-manager	1.37.15
	client-certificate-controller	1.37.15
	event-controller	1.37.15
	golang	1.20.4-alpine3.17
	kaas-public-api	1.37.15
	kaas-exporter	1.37.15
	kaas-ui	1.37.15
	license-controller	1.37.15
	lcm-controller	1.37.15
	machinepool-controller	1.37.15
	mcc-cache	1.37.15
	portforward-controller	1.37.15
	proxy-controller	1.37.15
	rbac-controller	1.37.15
	release-controller	1.37.15
	rhellicense-controller	1.37.15
	scope-controller	1.37.15
	user-controller	1.37.15
OpenStack ^Updated	openstack-provider	1.37.15
	os-credentials-controller	1.37.15
VMware vSphere ^Updated	vsphere-provider	1.37.15
	vsphere-credentials-controller	1.37.15
	keepalived	0.22.0-49-g9618f2a
	squid-proxy	0.0.1-10-g24a0d69

Artifacts¶

This section lists the component artifacts of the Container Cloud releases 2.24.0 - 2.24.2.

Bare metal artifacts
Core artifacts
IAM artifacts

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶
Artifact	Component	Path
Binaries	ironic-python-agent.initramfs ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20230606121129
	ironic-python-agent.kernel ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20230606121129
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-104-6e2e82c.tgz
Helm charts ^Updated	baremetal-api	https://binary.mirantis.com/core/helm/baremetal-api-1.37.15.tgz
	baremetal-operator	https://binary.mirantis.com/core/helm/baremetal-operator-1.37.15.tgz ^2.24.0
		https://binary.mirantis.com/core/helm/baremetal-operator-1.37.16.tgz ^2.24.1(2)
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.37.15.tgz
	baremetal-public-api	https://binary.mirantis.com/core/helm/baremetal-public-api-1.37.15.tgz
	kaas-ipam	https://binary.mirantis.com/core/helm/kaas-ipam-1.37.15.tgz
	local-volume-provisioner	https://binary.mirantis.com/core/helm/local-volume-provisioner-1.37.15.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.37.15.tgz
Docker images	ambasador ^Updated	mirantis.azurecr.io/core/external/nginx:1.37.15
	baremetal-dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-alpine-20230607171021
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-alpine-20230607164516 ^2.24.0
		mirantis.azurecr.io/bm/baremetal-operator:base-alpine-20230721153358 ^2.24.1(2)
	bm-collective ^Updated	mirantis.azurecr.io/bm/bm-collective:base-alpine-20230607154546
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.37.15
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:yoga-focal-20230605060019
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:yoga-focal-20230605060019
	ironic-prometheus-exporter ^Updated	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20230531081117
	kaas-ipam ^Updated	mirantis.azurecr.io/bm/kaas-ipam:base-alpine-20230614192933
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-27d64fb-20230421151539
	mariadb ^Updated	mirantis.azurecr.io/general/mariadb:10.6.12-focal-20230606052917
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.22.0-49-g9618f2a
	metallb-controller ^Updated	mirantis.azurecr.io/bm/external/metallb/controller:v0.13.9
	metallb-speaker ^Updated	mirantis.azurecr.io/bm/external/metallb/speaker:v0.13.9
	syslog-ng ^Updated	mirantis.azurecr.io/bm/syslog-ng:base-apline-20230607165607

Core artifacts¶
Artifact	Component	Paths
Bootstrap tarball ^Updated	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.37.15.tgz ^2.24.0(1)
		https://binary.mirantis.com/core/bin/bootstrap-darwin-1.37.19.tgz ^2.24.2
	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.37.15.tgz ^2.24.0(1)
		https://binary.mirantis.com/core/bin/bootstrap-linux-1.37.19.tgz ^2.24.2
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.37.15.tgz ^2.24.0
		https://binary.mirantis.com/core/helm/admission-controller-1.37.16.tgz ^2.24.1(2)
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.37.15.tgz
	byo-credentials-controller ^Removed	n/a
	byo-provider ^Removed	n/a
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.37.15.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.37.15.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.37.15.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.37.15.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.37.15.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.37.15.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.37.15.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.37.15.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.37.15.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.37.15.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.37.15.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.37.15.tgz
	mcc-cache-warmup ^New	https://binary.mirantis.com/core/helm/mcc-cache-warmup-1.37.15.tgz
	metrics-server	https://binary.mirantis.com/core/helm/metrics-server-1.37.15.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.37.15.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.37.15.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.37.15.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.37.15.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.37.15.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.37.15.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.37.15.tgz
	scope-controller	https://binary.mirantis.com/core/helm/scope-controller-1.37.15.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.37.15.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.37.15.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.37.15.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.37.15.tgz
	vsphere-vm-template-controller	https://binary.mirantis.com/core/helm/vsphere-vm-template-controller-1.37.15.tgz
Docker images ^Updated	admission-controller	mirantis.azurecr.io/core/admission-controller:1.37.15 ^2.24.0
		mirantis.azurecr.io/core/admission-controller:1.37.16 ^2.24.1(2)
	agent-controller	mirantis.azurecr.io/core/agent-controller:1.37.15
	byo-cluster-api-controller ^Removed	n/a
	byo-credentials-controller ^Removed	n/a
	ceph-kcc-controller	mirantis.azurecr.io/core/ceph-kcc-controller:1.37.15
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.11.0
	client-certificate-controller	mirantis.azurecr.io/core/client-certificate-controller:1.37.15
	event-controller	mirantis.azurecr.io/core/event-controller:1.37.15
	frontend	mirantis.azurecr.io/core/frontend:1.37.15
	iam-controller	mirantis.azurecr.io/core/iam-controller:1.37.15
	kaas-exporter	mirantis.azurecr.io/core/kaas-exporter:1.37.15
	kproxy	mirantis.azurecr.io/core/kproxy:1.37.15
	lcm-controller	mirantis.azurecr.io/core/lcm-controller:1.37.15
	license-controller	mirantis.azurecr.io/core/license-controller:1.37.15
	machinepool-controller	mirantis.azurecr.io/core/machinepool-controller:1.37.15
	mcc-haproxy	mirantis.azurecr.io/lcm/mcc-haproxy:v0.22.0-49-g9618f2a
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.22.0-49-g9618f2a
	metrics-server	mirantis.azurecr.io/lcm/metrics-server-amd64:v0.6.3-2
	nginx	mirantis.azurecr.io/core/external/nginx:1.37.15
	openstack-cloud-controller-manager	mirantis.azurecr.io/lcm/kubernetes/openstack-cloud-controller-manager-amd64:v1.24.5-10-g93314b86
	openstack-cluster-api-controller	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.37.15
	os-credentials-controller	mirantis.azurecr.io/core/os-credentials-controller:1.37.15
	portforward-controller	mirantis.azurecr.io/core/portforward-controller:1.37.15
	proxy-controller	mirantis.azurecr.io/core/proxy-controller:1.37.15
	rbac-controller	mirantis.azurecr.io/core/rbac-controller:1.37.15
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-4
	release-controller	mirantis.azurecr.io/core/release-controller:1.37.15
	rhellicense-controller	mirantis.azurecr.io/core/rhellicense-controller:1.37.15
	scope-controller	mirantis.azurecr.io/core/scope-controller:1.37.15
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-10-g24a0d69
	user-controller	mirantis.azurecr.io/core/user-controller:1.37.15
	vsphere-cluster-api-controller	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.37.15
	vsphere-credentials-controller	mirantis.azurecr.io/core/vsphere-credentials-controller:1.37.15
	vsphere-vm-template-controller	mirantis.azurecr.io/core/vsphere-vm-template-controller:1.37.15

IAM artifacts¶
Artifact	Component	Path
Binaries	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
Helm charts	iam ^Updated	https://binary.mirantis.com/iam/helm/iam-2.5.1.tgz ^2.24.0
		https://binary.mirantis.com/iam/helm/iam-2.5.3.tgz ^2.24.1(2)
Docker images ^Updated	keycloak	mirantis.azurecr.io/iam/keycloak:0.6.0
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-27d64fb-20230421151539
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.12-focal-20230423170220

Security notes¶

In total, since Container Cloud 2.23.0 major release, in 2.24.0, 2130 Common Vulnerabilities and Exposures (CVE) have been fixed: 98 of critical and 2032 of high severity.

Among them, 984 CVEs that are listed in Addressed CVEs - detailed Addressed CVEs - detailed have been fixed since the 2.23.5 patch release: 62 of critical and 922 of high severity. The remaining CVEs were addressed since Container Cloud 2.23.0 and the fixes released with the patch releases of the 2.23.x series.

The summary table contains the total number of unique CVEs along with the total number of issues fixed across the images.

The full list of the CVEs present in the current Container Cloud release is available at the Mirantis Security Portal.

Addressed CVEs - summary¶
Severity	Critical	High	Total
Unique CVEs	18	88	106
Total issues across images	62	922	984

Addressed CVEs - detailed¶
Image	Component name	CVE
bm/baremetal-dnsmasq	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
bm/baremetal-operator	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
	cryptography	CVE-2023-2650 (High)
bm/bm-collective	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
bm/kaas-ipam	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
bm/syslog-ng	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
	ncurses-libs	CVE-2023-29491 (High)
	ncurses-terminfo-base	CVE-2023-29491 (High)
ceph/mcp/ceph-controller	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
ceph/rook	openssl	CVE-2022-3786 (High)
		CVE-2023-0286 (High)
		CVE-2022-3602 (High)
	openssl-libs	CVE-2022-3602 (High)
		CVE-2022-3786 (High)
		CVE-2023-0286 (High)
	cryptography	CVE-2023-2650 (High)
core/admission-controller	helm.sh/helm/v3	CVE-2021-32690 (High)
		CVE-2022-23525 (High)
		CVE-2022-23526 (High)
		CVE-2022-23524 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/agent-controller	helm.sh/helm/v3	CVE-2021-32690 (High)
		CVE-2022-23525 (High)
		CVE-2022-23526 (High)
		CVE-2022-23524 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/aws-cluster-api-controller	helm.sh/helm/v3	CVE-2021-32690 (High)
		CVE-2022-23525 (High)
		CVE-2022-23526 (High)
		CVE-2022-23524 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/aws-credentials-controller	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/azure-cluster-api-controller	helm.sh/helm/v3	CVE-2022-23525 (High)
		CVE-2022-23526 (High)
		CVE-2022-23524 (High)
		CVE-2021-32690 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/azure-credentials-controller	helm.sh/helm/v3	CVE-2021-32690 (High)
		CVE-2022-23525 (High)
		CVE-2022-23526 (High)
		CVE-2022-23524 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/bootstrap-controller	helm.sh/helm/v3	CVE-2022-23525 (High)
		CVE-2022-23526 (High)
		CVE-2022-23524 (High)
		CVE-2021-32690 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/byo-cluster-api-controller	helm.sh/helm/v3	CVE-2021-32690 (High)
		CVE-2022-23525 (High)
		CVE-2022-23526 (High)
		CVE-2022-23524 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/byo-credentials-controller	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/ceph-kcc-controller	helm.sh/helm/v3	CVE-2022-23525 (High)
		CVE-2022-23526 (High)
		CVE-2022-23524 (High)
		CVE-2021-32690 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/cluster-api-provider-baremetal	helm.sh/helm/v3	CVE-2022-23525 (High)
		CVE-2022-23526 (High)
		CVE-2022-23524 (High)
		CVE-2021-32690 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/configuration-collector	helm.sh/helm/v3	CVE-2021-32690 (High)
		CVE-2022-23525 (High)
		CVE-2022-23526 (High)
		CVE-2022-23524 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/equinix-cluster-api-controller	helm.sh/helm/v3	CVE-2021-32690 (High)
		CVE-2022-23525 (High)
		CVE-2022-23526 (High)
		CVE-2022-23524 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/equinix-credentials-controller	helm.sh/helm/v3	CVE-2021-32690 (High)
		CVE-2022-23525 (High)
		CVE-2022-23526 (High)
		CVE-2022-23524 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/event-controller	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/external/nginx	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
	libx11	CVE-2023-3138 (High)
	ncurses-libs	CVE-2023-29491 (High)
	ncurses-terminfo-base	CVE-2023-29491 (High)
core/frontend	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
	libx11	CVE-2023-3138 (High)
	ncurses-libs	CVE-2023-29491 (High)
	ncurses-terminfo-base	CVE-2023-29491 (High)
core/iam-controller	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/kaas-exporter	helm.sh/helm/v3	CVE-2021-32690 (High)
		CVE-2022-23525 (High)
		CVE-2022-23526 (High)
		CVE-2022-23524 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/kproxy	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/lcm-controller	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/license-controller	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/machinepool-controller	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/openstack-cluster-api-controller	helm.sh/helm/v3	CVE-2022-23525 (High)
		CVE-2022-23526 (High)
		CVE-2022-23524 (High)
		CVE-2021-32690 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/os-credentials-controller	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/portforward-controller	helm.sh/helm/v3	CVE-2022-23525 (High)
		CVE-2022-23526 (High)
		CVE-2022-23524 (High)
		CVE-2021-32690 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/proxy-controller	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/rbac-controller	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/release-controller	helm.sh/helm/v3	CVE-2021-32690 (High)
		CVE-2022-23525 (High)
		CVE-2022-23526 (High)
		CVE-2022-23524 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/rhellicense-controller	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/scope-controller	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/user-controller	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/vsphere-cluster-api-controller	helm.sh/helm/v3	CVE-2022-23525 (High)
		CVE-2022-23526 (High)
		CVE-2022-23524 (High)
		CVE-2021-32690 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/vsphere-credentials-controller	helm.sh/helm/v3	CVE-2022-23525 (High)
		CVE-2022-23526 (High)
		CVE-2022-23524 (High)
		CVE-2021-32690 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
core/vsphere-vm-template-controller	helm.sh/helm/v3	CVE-2021-32690 (High)
		CVE-2022-23525 (High)
		CVE-2022-23526 (High)
		CVE-2022-23524 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
iam/keycloak	io.vertx:vertx-core	CVE-2021-4125 (High)
		CVE-2021-44228 (Critical)
		CVE-2021-44530 (Critical)
		CVE-2021-45046 (Critical)
	org.apache.cxf:cxf-core	CVE-2022-46364 (Critical)
		CVE-2022-46363 (High)
	org.apache.cxf:cxf-rt-transports-http	CVE-2022-46363 (High)
		CVE-2022-46364 (Critical)
	org.apache.santuario:xmlsec	CVE-2022-21476 (High)
		CVE-2022-47966 (Critical)
	org.apache.kafka:kafka-clients	CVE-2023-25194 (High)
		CVE-2021-46877 (High)
		CVE-2020-36518 (High)
	com.fasterxml.jackson.core:jackson-databind	CVE-2023-35116 (High)
		CVE-2022-42003 (High)
		CVE-2022-42004 (High)
		CVE-2023-35116 (High)
		CVE-2022-42003 (High)
		CVE-2022-42004 (High)
		CVE-2023-35116 (High)
		CVE-2022-42003 (High)
		CVE-2022-42004 (High)
	com.google.protobuf:protobuf-java	CVE-2022-3509 (High)
		CVE-2022-3510 (High)
	com.google.protobuf:protobuf-java-util	CVE-2022-3509 (High)
		CVE-2022-3510 (High)
	org.yaml:snakeyaml	CVE-2022-25857 (High)
	java-11-openjdk-headless	CVE-2023-21930 (High)
	platform-python	CVE-2023-24329 (High)
	python3-libs	CVE-2023-24329 (High)
lcm/docker/ucp	curl	CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-23914 (Critical)
		CVE-2023-28319 (High)
	libcurl	CVE-2023-28319 (High)
		CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-23914 (Critical)
	github.com/crewjam/saml	CVE-2022-41912 (Critical)
		CVE-2023-28119 (High)
	libcrypto1.1	CVE-2023-0464 (High)
		CVE-2023-2650 (High)
	libssl1.1	CVE-2023-0464 (High)
		CVE-2023-2650 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	github.com/opencontainers/runc	CVE-2023-28642 (High)
	github.com/docker/cli	CVE-2021-41092 (High)
	golang.org/x/net	CVE-2022-27664 (High)
	golang.org/x/text	CVE-2022-32149 (High)
	ncurses-libs	CVE-2023-29491 (High)
	ncurses-terminfo-base	CVE-2023-29491 (High)
lcm/docker/ucp-agent	curl	CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-28319 (High)
		CVE-2023-23914 (Critical)
	libcurl	CVE-2023-23914 (Critical)
		CVE-2023-28319 (High)
		CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
	github.com/crewjam/saml	CVE-2022-41912 (Critical)
		CVE-2023-28119 (High)
	libcrypto1.1	CVE-2023-2650 (High)
		CVE-2023-0464 (High)
		CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
	libssl1.1	CVE-2023-0464 (High)
		CVE-2023-2650 (High)
		CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	github.com/opencontainers/runc	CVE-2023-28642 (High)
	github.com/docker/cli	CVE-2021-41092 (High)
	golang.org/x/net	CVE-2022-27664 (High)
	golang.org/x/text	CVE-2022-32149 (High)
lcm/docker/ucp-auth	curl	CVE-2023-23914 (Critical)
		CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-28319 (High)
	libcurl	CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-28319 (High)
		CVE-2023-23914 (Critical)
	github.com/crewjam/saml	CVE-2022-41912 (Critical)
		CVE-2023-28119 (High)
	libcrypto1.1	CVE-2023-0464 (High)
		CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
		CVE-2023-2650 (High)
	libssl1.1	CVE-2023-2650 (High)
		CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
		CVE-2023-0464 (High)
	golang.org/x/net	CVE-2022-27664 (High)
	golang.org/x/text	CVE-2022-32149 (High)
lcm/docker/ucp-auth-store	github.com/crewjam/saml	CVE-2023-28119 (High)
		CVE-2022-41912 (Critical)
	curl	CVE-2023-28319 (High)
		CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
	libcurl	CVE-2023-28319 (High)
		CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
	libcrypto1.1	CVE-2023-2650 (High)
		CVE-2023-0464 (High)
	libssl1.1	CVE-2023-0464 (High)
		CVE-2023-2650 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	golang.org/x/net	CVE-2022-27664 (High)
	golang.org/x/text	CVE-2022-32149 (High)
	ncurses-libs	CVE-2023-29491 (High)
	ncurses-terminfo-base	CVE-2023-29491 (High)
lcm/docker/ucp-azure-ip-allocator	curl	CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-28319 (High)
	libcurl	CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-28319 (High)
	libcrypto1.1	CVE-2023-2650 (High)
		CVE-2023-0464 (High)
	libssl1.1	CVE-2023-2650 (High)
		CVE-2023-0464 (High)
	ncurses-libs	CVE-2023-29491 (High)
	ncurses-terminfo-base	CVE-2023-29491 (High)
lcm/docker/ucp-calico-cni	github.com/emicklei/go-restful	CVE-2022-1996 (Critical)
	golang.org/x/crypto	CVE-2022-27191 (High)
		CVE-2020-29652 (High)
		CVE-2021-43565 (High)
	golang.org/x/text	CVE-2022-32149 (High)
		CVE-2020-14040 (High)
		CVE-2021-38561 (High)
		CVE-2022-32149 (High)
	golang.org/x/net	CVE-2022-27664 (High)
		CVE-2021-33194 (High)
		CVE-2022-27664 (High)
	github.com/containernetworking/cni	CVE-2021-20206 (High)
	github.com/gogo/protobuf	CVE-2021-3121 (High)
lcm/docker/ucp-calico-kube-controllers	github.com/emicklei/go-restful	CVE-2022-1996 (Critical)
	golang.org/x/net	CVE-2022-27664 (High)
	golang.org/x/text	CVE-2022-32149 (High)
lcm/docker/ucp-calico-node	github.com/emicklei/go-restful	CVE-2022-1996 (Critical)
	openssl-libs	CVE-2023-0286 (High)
	golang.org/x/net	CVE-2022-27664 (High)
	golang.org/x/text	CVE-2022-32149 (High)
lcm/docker/ucp-cfssl	curl	CVE-2023-28319 (High)
		CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-23914 (Critical)
	libcurl	CVE-2023-23914 (Critical)
		CVE-2023-28319 (High)
		CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
	libcrypto1.1	CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
		CVE-2023-0464 (High)
		CVE-2023-2650 (High)
	libssl1.1	CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
		CVE-2023-0464 (High)
		CVE-2023-2650 (High)
	golang.org/x/net	CVE-2022-27664 (High)
	golang.org/x/text	CVE-2022-32149 (High)
lcm/docker/ucp-compose	github.com/emicklei/go-restful	CVE-2022-1996 (Critical)
	golang.org/x/crypto	CVE-2021-43565 (High)
		CVE-2022-27191 (High)
		CVE-2021-43565 (High)
		CVE-2022-27191 (High)
	golang.org/x/net	CVE-2021-33194 (High)
		CVE-2022-27664 (High)
		CVE-2021-33194 (High)
		CVE-2022-27664 (High)
	golang.org/x/text	CVE-2022-32149 (High)
		CVE-2021-38561 (High)
		CVE-2022-32149 (High)
		CVE-2021-38561 (High)
		CVE-2022-32149 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	github.com/prometheus/client_golang	CVE-2022-21698 (High)
lcm/docker/ucp-containerd-shim-process	golang.org/x/net	CVE-2021-33194 (High)
		CVE-2022-27664 (High)
		CVE-2021-33194 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
lcm/docker/ucp-controller	curl	CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-28319 (High)
		CVE-2023-23914 (Critical)
	libcurl	CVE-2023-23914 (Critical)
		CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-28319 (High)
	github.com/crewjam/saml	CVE-2022-41912 (Critical)
		CVE-2023-28119 (High)
	libcrypto1.1	CVE-2023-2650 (High)
		CVE-2023-0464 (High)
		CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
	libssl1.1	CVE-2023-2650 (High)
		CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
		CVE-2023-0464 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	github.com/opencontainers/runc	CVE-2023-28642 (High)
	github.com/docker/cli	CVE-2021-41092 (High)
	golang.org/x/net	CVE-2022-27664 (High)
	golang.org/x/text	CVE-2022-32149 (High)
lcm/docker/ucp-coredns	golang.org/x/net	CVE-2022-27664 (High)
		CVE-2022-41721 (High)
	golang.org/x/text	CVE-2022-32149 (High)
lcm/docker/ucp-dsinfo	github.com/emicklei/go-restful	CVE-2022-1996 (Critical)
	golang.org/x/crypto	CVE-2021-43565 (High)
		CVE-2022-27191 (High)
		CVE-2021-43565 (High)
	golang.org/x/net	CVE-2022-27664 (High)
		CVE-2021-33194 (High)
		CVE-2022-27664 (High)
	golang.org/x/text	CVE-2022-32149 (High)
		CVE-2021-38561 (High)
		CVE-2022-32149 (High)
		CVE-2021-38561 (High)
		CVE-2022-32149 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	github.com/prometheus/client_golang	CVE-2022-21698 (High)
lcm/docker/ucp-etcd	curl	CVE-2023-28319 (High)
		CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-23914 (Critical)
	libcurl	CVE-2023-28319 (High)
		CVE-2023-23914 (Critical)
		CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
	libcrypto1.1	CVE-2023-2650 (High)
		CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
		CVE-2023-0464 (High)
	libssl1.1	CVE-2023-0464 (High)
		CVE-2023-2650 (High)
		CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
	golang.org/x/text	CVE-2022-32149 (High)
		CVE-2021-38561 (High)
		CVE-2022-32149 (High)
		CVE-2021-38561 (High)
		CVE-2022-32149 (High)
	golang.org/x/net	CVE-2022-27664 (High)
lcm/docker/ucp-hardware-info	curl	CVE-2023-28319 (High)
		CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-23914 (Critical)
	libcurl	CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-23914 (Critical)
		CVE-2023-28319 (High)
	libcrypto1.1	CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
		CVE-2023-2650 (High)
		CVE-2023-0464 (High)
	libssl1.1	CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
		CVE-2023-2650 (High)
		CVE-2023-0464 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	golang.org/x/net	CVE-2022-27664 (High)
	golang.org/x/text	CVE-2022-32149 (High)
lcm/docker/ucp-interlock	curl	CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-28319 (High)
		CVE-2023-23914 (Critical)
	libcurl	CVE-2023-28319 (High)
		CVE-2023-23914 (Critical)
		CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
	libcrypto1.1	CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
		CVE-2023-2650 (High)
		CVE-2023-0464 (High)
	libssl1.1	CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
		CVE-2023-2650 (High)
		CVE-2023-0464 (High)
	golang.org/x/net	CVE-2022-41721 (High)
		CVE-2022-27664 (High)
	github.com/containerd/containerd	CVE-2023-25173 (High)
	golang.org/x/text	CVE-2022-32149 (High)
lcm/docker/ucp-interlock-config	curl	CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-28319 (High)
	libcurl	CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-28319 (High)
	libcrypto1.1	CVE-2023-2650 (High)
		CVE-2023-0464 (High)
	libssl1.1	CVE-2023-2650 (High)
		CVE-2023-0464 (High)
	libwebp	CVE-2023-1999 (High)
	ncurses-libs	CVE-2023-29491 (High)
	ncurses-terminfo-base	CVE-2023-29491 (High)
lcm/docker/ucp-interlock-extension	curl	CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-28319 (High)
		CVE-2023-23914 (Critical)
	libcurl	CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-23914 (Critical)
		CVE-2023-28319 (High)
	libcrypto1.1	CVE-2023-2650 (High)
		CVE-2023-0464 (High)
		CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
	libssl1.1	CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
		CVE-2023-0464 (High)
		CVE-2023-2650 (High)
	golang.org/x/net	CVE-2022-41721 (High)
		CVE-2022-27664 (High)
	golang.org/x/text	CVE-2022-32149 (High)
lcm/docker/ucp-interlock-proxy	curl	CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-28319 (High)
	libcurl	CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-28319 (High)
	libcrypto1.1	CVE-2023-2650 (High)
		CVE-2023-0464 (High)
	libssl1.1	CVE-2023-0464 (High)
		CVE-2023-2650 (High)
	libwebp	CVE-2023-1999 (High)
	ncurses-libs	CVE-2023-29491 (High)
	ncurses-terminfo-base	CVE-2023-29491 (High)
lcm/docker/ucp-kube-ingress-controller	curl	CVE-2022-43551 (High)
		CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-23914 (Critical)
		CVE-2022-32221 (Critical)
		CVE-2022-42915 (High)
		CVE-2022-42916 (High)
		CVE-2023-28319 (High)
	libcurl	CVE-2022-32221 (Critical)
		CVE-2022-42915 (High)
		CVE-2022-42916 (High)
		CVE-2023-23914 (Critical)
		CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-28319 (High)
		CVE-2022-43551 (High)
	libcrypto1.1	CVE-2023-0464 (High)
		CVE-2023-2650 (High)
		CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
	libssl1.1	CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
		CVE-2023-0464 (High)
		CVE-2023-2650 (High)
	openssl	CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
		CVE-2023-2650 (High)
		CVE-2023-0464 (High)
	golang.org/x/net	CVE-2022-41721 (High)
		CVE-2022-27664 (High)
	libxml2	CVE-2022-40303 (High)
		CVE-2022-40304 (High)
	github.com/opencontainers/runc	CVE-2023-28642 (High)
	golang.org/x/text	CVE-2022-32149 (High)
	ncurses-libs	CVE-2023-29491 (High)
	ncurses-terminfo-base	CVE-2023-29491 (High)
lcm/docker/ucp-metrics	curl	CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-28319 (High)
	libcurl	CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-28319 (High)
	libcrypto1.1	CVE-2023-0464 (High)
		CVE-2023-2650 (High)
	libssl1.1	CVE-2023-2650 (High)
		CVE-2023-0464 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	golang.org/x/net	CVE-2022-41723 (High)
lcm/docker/ucp-node-feature-discovery	libssl3	CVE-2023-0286 (High)
	openssl	CVE-2023-0286 (High)
	github.com/prometheus/client_golang	CVE-2022-21698 (High)
	golang.org/x/net	CVE-2022-27664 (High)
	golang.org/x/text	CVE-2022-32149 (High)
	gopkg.in/yaml.v3	CVE-2022-28948 (High)
lcm/docker/ucp-nvidia-device-plugin	golang.org/x/net	CVE-2022-27664 (High)
		CVE-2021-33194 (High)
	golang.org/x/text	CVE-2022-32149 (High)
		CVE-2021-38561 (High)
	libssl3	CVE-2023-0286 (High)
	openssl	CVE-2023-0286 (High)
	github.com/prometheus/client_golang	CVE-2022-21698 (High)
lcm/docker/ucp-nvidia-gpu-feature-discovery	golang.org/x/net	CVE-2022-41721 (High)
		CVE-2022-27664 (High)
	libssl3	CVE-2023-0286 (High)
	openssl	CVE-2023-0286 (High)
	golang.org/x/text	CVE-2022-32149 (High)
lcm/docker/ucp-secureoverlay-agent	curl	CVE-2023-28319 (High)
		CVE-2023-23914 (Critical)
		CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
	libcurl	CVE-2023-28319 (High)
		CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-23914 (Critical)
	libcrypto1.1	CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
		CVE-2023-0464 (High)
		CVE-2023-2650 (High)
	libssl1.1	CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
		CVE-2023-0464 (High)
		CVE-2023-2650 (High)
	golang.org/x/net	CVE-2022-27664 (High)
	golang.org/x/text	CVE-2022-32149 (High)
lcm/docker/ucp-secureoverlay-mgr	curl	CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-23914 (Critical)
		CVE-2023-28319 (High)
	libcurl	CVE-2023-23914 (Critical)
		CVE-2023-28319 (High)
		CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
	libcrypto1.1	CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
		CVE-2023-0464 (High)
		CVE-2023-2650 (High)
	libssl1.1	CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
		CVE-2023-0464 (High)
		CVE-2023-2650 (High)
	golang.org/x/net	CVE-2022-27664 (High)
	golang.org/x/text	CVE-2022-32149 (High)
lcm/docker/ucp-sf-notifier	Werkzeug	CVE-2022-29361 (Critical)
		CVE-2023-25577 (High)
	libcrypto1.1	CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
		CVE-2023-0464 (High)
		CVE-2023-2650 (High)
	libssl1.1	CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
		CVE-2023-2650 (High)
		CVE-2023-0464 (High)
	openssl-dev	CVE-2023-0464 (High)
		CVE-2023-2650 (High)
		CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
	cryptography	CVE-2023-2650 (High)
	Flask	CVE-2023-30861 (High)
	krb5-libs	CVE-2022-42898 (High)
	ncurses-libs	CVE-2023-29491 (High)
	ncurses-terminfo-base	CVE-2023-29491 (High)
	wheel	CVE-2022-40898 (High)
lcm/docker/ucp-swarm	curl	CVE-2023-23914 (Critical)
		CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-28319 (High)
	libcurl	CVE-2023-27533 (High)
		CVE-2023-27534 (High)
		CVE-2023-27536 (High)
		CVE-2023-28319 (High)
		CVE-2023-23914 (Critical)
	libcrypto1.1	CVE-2023-0464 (High)
		CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
		CVE-2023-2650 (High)
	libssl1.1	CVE-2023-0464 (High)
		CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
		CVE-2023-2650 (High)
	github.com/hashicorp/consul	CVE-2022-29153 (High)
		CVE-2022-38149 (High)
		CVE-2020-7219 (High)
		CVE-2021-37219 (High)
	golang.org/x/crypto	CVE-2022-27191 (High)
		CVE-2020-29652 (High)
		CVE-2021-43565 (High)
	golang.org/x/net	CVE-2021-33194 (High)
		CVE-2022-27664 (High)
	github.com/docker/docker	CVE-2023-28840 (High)
	github.com/docker/distribution	CVE-2017-11468 (High)
lcm/external/aws-cloud-controller-manager	github.com/emicklei/go-restful	CVE-2022-1996 (Critical)
	golang.org/x/crypto	CVE-2021-43565 (High)
		CVE-2022-27191 (High)
	github.com/prometheus/client_golang	CVE-2022-21698 (High)
	golang.org/x/net	CVE-2022-27664 (High)
	golang.org/x/text	CVE-2022-32149 (High)
	gopkg.in/yaml.v3	CVE-2022-28948 (High)
lcm/external/aws-ebs-csi-driver	ncurses-libs	CVE-2023-29491 (High)
	systemd-libs	CVE-2023-26604 (High)
	golang.org/x/net	CVE-2022-41721 (High)
	golang.org/x/text	CVE-2022-32149 (High)
lcm/external/csi-attacher	golang.org/x/crypto	CVE-2021-43565 (High)
		CVE-2022-27191 (High)
		CVE-2020-29652 (High)
		CVE-2021-43565 (High)
		CVE-2022-27191 (High)
		CVE-2020-29652 (High)
		CVE-2021-43565 (High)
		CVE-2022-27191 (High)
		CVE-2020-29652 (High)
	golang.org/x/net	CVE-2021-33194 (High)
	golang.org/x/text	CVE-2021-38561 (High)
	github.com/gogo/protobuf	CVE-2021-3121 (High)
	github.com/emicklei/go-restful	CVE-2022-1996 (Critical)
lcm/external/csi-provisioner	github.com/emicklei/go-restful	CVE-2022-1996 (Critical)
lcm/external/csi-resizer	github.com/emicklei/go-restful	CVE-2022-1996 (Critical)
lcm/helm/tiller	libcrypto1.1	CVE-2021-23840 (High)
		CVE-2020-1967 (High)
		CVE-2021-3450 (High)
		CVE-2021-3711 (Critical)
		CVE-2021-3712 (High)
	libssl1.1	CVE-2020-1967 (High)
		CVE-2021-3450 (High)
		CVE-2021-3711 (Critical)
		CVE-2021-3712 (High)
		CVE-2021-23840 (High)
	apk-tools	CVE-2021-36159 (Critical)
		CVE-2021-30139 (High)
	zlib	CVE-2022-37434 (Critical)
	busybox	CVE-2021-42378 (High)
		CVE-2021-42379 (High)
		CVE-2021-42380 (High)
		CVE-2021-42381 (High)
		CVE-2021-42382 (High)
		CVE-2021-42383 (High)
		CVE-2021-42384 (High)
		CVE-2021-42385 (High)
		CVE-2021-42386 (High)
		CVE-2021-28831 (High)
	ssl_client	CVE-2021-28831 (High)
		CVE-2021-42378 (High)
		CVE-2021-42379 (High)
		CVE-2021-42380 (High)
		CVE-2021-42381 (High)
		CVE-2021-42382 (High)
		CVE-2021-42383 (High)
		CVE-2021-42384 (High)
		CVE-2021-42385 (High)
		CVE-2021-42386 (High)
lcm/kubernetes/cinder-csi-plugin-amd64	libtasn1-6	CVE-2021-46848 (Critical)
	github.com/emicklei/go-restful	CVE-2022-1996 (Critical)
	libssl1.1	CVE-2023-0286 (High)
		CVE-2022-4450 (High)
		CVE-2023-0215 (High)
	openssl	CVE-2023-0286 (High)
		CVE-2022-4450 (High)
		CVE-2023-0215 (High)
	libsystemd0	CVE-2023-26604 (High)
	libudev1	CVE-2023-26604 (High)
	udev	CVE-2023-26604 (High)
	libgnutls30	CVE-2023-0361 (High)
	golang.org/x/net	CVE-2022-27664 (High)
	golang.org/x/text	CVE-2022-32149 (High)
	gopkg.in/yaml.v3	CVE-2022-28948 (High)
lcm/kubernetes/openstack-cloud-controller-manager-amd64	github.com/emicklei/go-restful	CVE-2022-1996 (Critical)
	zlib	CVE-2022-37434 (Critical)
	golang.org/x/crypto	CVE-2022-27191 (High)
		CVE-2021-43565 (High)
	golang.org/x/text	CVE-2021-38561 (High)
		CVE-2022-32149 (High)
	github.com/prometheus/client_golang	CVE-2022-21698 (High)
	golang.org/x/net	CVE-2022-27664 (High)
	gopkg.in/yaml.v3	CVE-2022-28948 (High)
	k8s.io/kubernetes	CVE-2021-25741 (High)
lcm/mcc-haproxy	pcre2	CVE-2022-1586 (Critical)
		CVE-2022-1587 (Critical)
	zlib	CVE-2022-37434 (Critical)
	libcrypto1.1	CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
		CVE-2023-2650 (High)
		CVE-2023-0464 (High)
	libssl1.1	CVE-2022-4450 (High)
		CVE-2023-0215 (High)
		CVE-2023-0286 (High)
		CVE-2023-2650 (High)
		CVE-2023-0464 (High)
	busybox	CVE-2022-30065 (High)
	ssl_client	CVE-2022-30065 (High)
lcm/registry	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
mirantis/ceph	openssl	CVE-2022-3786 (High)
		CVE-2023-0286 (High)
		CVE-2022-3602 (High)
	openssl-libs	CVE-2022-3602 (High)
		CVE-2022-3786 (High)
		CVE-2023-0286 (High)
	python3	CVE-2023-24329 (High)
	python3-devel	CVE-2023-24329 (High)
	python3-libs	CVE-2023-24329 (High)
mirantis/cephcsi	openssl	CVE-2022-3786 (High)
		CVE-2023-0286 (High)
		CVE-2022-3602 (High)
	openssl-libs	CVE-2022-3602 (High)
		CVE-2022-3786 (High)
		CVE-2023-0286 (High)
	cryptography	CVE-2023-2650 (High)
mirantis/fio	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
stacklight/alerta-web	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
stacklight/alertmanager-webhook-servicenow	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
	openssl-dev	CVE-2023-2650 (High)
	Flask	CVE-2023-30861 (High)
stacklight/alpine-utils	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
	ncurses-libs	CVE-2023-29491 (High)
	ncurses-terminfo-base	CVE-2023-29491 (High)
stacklight/blackbox-exporter	golang.org/x/net	CVE-2022-41723 (High)
stacklight/cadvisor	libcrypto1.1	CVE-2023-2650 (High)
	libssl1.1	CVE-2023-2650 (High)
stacklight/cerebro	org.xerial:sqlite-jdbc	CVE-2023-32697 (Critical)
	com.fasterxml.jackson.core:jackson-databind	CVE-2023-35116 (High)
		CVE-2022-42003 (High)
		CVE-2022-42004 (High)
		CVE-2020-36518 (High)
		CVE-2021-46877 (High)
	libssl1.1	CVE-2023-2650 (High)
		CVE-2023-0464 (High)
	openssl	CVE-2023-2650 (High)
		CVE-2023-0464 (High)
stacklight/ironic-prometheus-exporter	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
stacklight/k8s-sidecar	libcrypto1.1	CVE-2023-2650 (High)
	libssl1.1	CVE-2023-2650 (High)
	ncurses-libs	CVE-2023-29491 (High)
	ncurses-terminfo-base	CVE-2023-29491 (High)
stacklight/kubectl	libssl1.1	CVE-2023-2650 (High)
		CVE-2023-0464 (High)
	openssl	CVE-2023-2650 (High)
		CVE-2023-0464 (High)
stacklight/metric-collector	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
stacklight/node-exporter	golang.org/x/net	CVE-2022-41723 (High)
stacklight/opensearch	org.codelibs.elasticsearch.module:ingest-common	CVE-2019-7611 (High)
		CVE-2015-5377 (Critical)
	org.springframework:spring-core	CVE-2023-20860 (High)
stacklight/opensearch-dashboards	decode-uri-component	CVE-2022-38900 (High)
	glob-parent	CVE-2021-35065 (High)
stacklight/prometheus	github.com/docker/docker	CVE-2023-28840 (High)
	golang.org/x/net	CVE-2022-41723 (High)
stacklight/prometheus-es-exporter	libcrypto1.1	CVE-2023-2650 (High)
	libssl1.1	CVE-2023-2650 (High)
	ncurses-libs	CVE-2023-29491 (High)
	ncurses-terminfo-base	CVE-2023-29491 (High)
stacklight/prometheus-libvirt-exporter	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
stacklight/prometheus-patroni-exporter	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
stacklight/prometheus-relay	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
stacklight/sf-notifier	libcrypto1.1	CVE-2023-2650 (High)
	libssl1.1	CVE-2023-2650 (High)
	ncurses-libs	CVE-2023-29491 (High)
	ncurses-terminfo-base	CVE-2023-29491 (High)
	openssl-dev	CVE-2023-2650 (High)
stacklight/sf-reporter	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
	ncurses-libs	CVE-2023-29491 (High)
	ncurses-terminfo-base	CVE-2023-29491 (High)
stacklight/stacklight-toolkit	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
	ncurses-libs	CVE-2023-29491 (High)
	ncurses-terminfo-base	CVE-2023-29491 (High)
stacklight/telegraf	libssl1.1	CVE-2023-2650 (High)
		CVE-2023-0464 (High)
		CVE-2023-2650 (High)
		CVE-2023-0464 (High)
	openssl	CVE-2023-2650 (High)
		CVE-2023-0464 (High)
		CVE-2023-2650 (High)
		CVE-2023-0464 (High)
stacklight/telemeter	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
stacklight/tungstenfabric-prometheus-exporter	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)
stacklight/yq	libcrypto3	CVE-2023-2650 (High)
	libssl3	CVE-2023-2650 (High)

Update notes¶

This section describes the specific actions you as a cloud operator need to complete before or after your Container Cloud cluster update to the Cluster release 14.0.0.

Consider this information as a supplement to the generic update procedures published in Operations Guide: Automatic upgrade of a management cluster and Update a managed cluster.

Pre-update actions¶

Update L2 templates on existing bare metal clusters¶

Since Container Cloud 2.24.0, the use of the l3Layout section in L2 templates is mandatory. Therefore, if your L2 templates do not contain this section, manually add it for all existing clusters by defining all subnets that are used in the npTemplate section of the L2 template.

For details on L2 template configuration, see Create L2 templates.

Caution

Partial definition of subnets is prohibited.

See also

2.23.5¶

Container Cloud 2.23.5 is the fourth patch release of the 2.23.x release series that incorporates security fixes for CVEs of Critical and High severity. This patch release:

Introduces the patch Cluster release 12.7.4 for MOSK 23.1.4.
Introduces the patch Cluster release 11.7.4
Supports the latest major Cluster releases 12.7.0, 11.7.0.
Does not support greenfield deployments based on deprecated Cluster releases 12.7.3, 11.7.3, 12.7.2, 11.7.2, 12.7.1, 11.7.1, 12.5.0, and 11.6.0. Use the latest available Cluster releases of the series instead.

This section describes known issues and contains the lists of updated artifacts and CVE fixes for the Container Cloud release 2.23.5. For CVE fixes delivered with the previous patch release, see security notes for 2.23.4, 2.23.3, and 2.23.2.

For enhancements, addressed and known issues of the parent Container Cloud release 2.23.0, refer to 2.23.0.

Artifacts¶

This section lists the components artifacts of the Container Cloud patch release 2.23.5. For artifacts of the Cluster releases introduced in 2.23.5, see Cluster release 12.7.4 and Cluster release 11.7.4.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries	baremetal-api ^Updated	https://binary.mirantis.com/core/helm/baremetal-api-1.36.27.tgz
	baremetal-operator ^Updated	https://binary.mirantis.com/core/helm/baremetal-operator-1.36.27.tgz
	baremetal-public-api ^Updated	https://binary.mirantis.com/core/helm/baremetal-public-api-1.36.27.tgz
	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20230126190304
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20230126190304
	kaas-ipam ^Updated	https://binary.mirantis.com/core/helm/kaas-ipam-1.36.27.tgz
	local-volume-provisioner ^Updated	https://binary.mirantis.com/core/helm/local-volume-provisioner-1.36.27.tgz
	metallb ^Updated	https://binary.mirantis.com/core/helm/metallb-1.36.27.tgz
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-104-6e2e82c.tgz
Docker images	ambassador ^Updated	mirantis.azurecr.io/core/external/nginx:1.36.27
	baremetal-dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-alpine-20230522161215
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-focal-20230522160916
	bm-collective ^Updated	mirantis.azurecr.io/bm/bm-collective:base-alpine-20230522161437
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:yoga-focal-20230523063451
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:yoga-focal-20230523063451
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20230330140456
	kaas-ipam ^Updated	mirantis.azurecr.io/bm/kaas-ipam:base-focal-20230522161025
	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.1-27d64fb-20230421151539
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.12-focal-20230423170220
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.19.0-5-g6a7e17d
	metallb-controller ^Updated	mirantis.azurecr.io/bm/external/metallb/controller:v0.13.7-3
	metallb-speaker ^Updated	mirantis.azurecr.io/bm/external/metallb/speaker:v0.13.7-3
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20230424092635

Core artifacts¶

Artifact	Component	Paths
Bootstrap tarball ^Updated	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.36.28.tar.gz
	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.36.28.tar.gz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.36.27.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.36.27.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.36.27.tgz
	byo-credentials-controller	https://binary.mirantis.com/core/helm/byo-credentials-controller-1.36.27.tgz
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.36.27.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.36.27.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.36.27.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.36.27.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.36.27.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.36.27.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.36.27.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.36.27.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.36.27.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.36.27.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.36.27.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.36.27.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.36.27.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.36.27.tgz
	metrics-server	https://binary.mirantis.com/core/helm/metrics-server-1.36.27.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.36.27.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.36.27.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.36.27.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.36.27.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.36.27.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.36.27.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.36.27.tgz
	scope-controller	http://binary.mirantis.com/core/helm/scope-controller-1.36.27.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.36.27.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.36.27.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.36.27.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.36.27.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.36.27
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.36.27
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.36.27
	byo-credentials-controller ^Updated	mirantis.azurecr.io/core/byo-credentials-controller:1.36.27
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.36.27
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.6.1
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.36.27
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.36.27
	configuration-collector ^Updated	mirantis.azurecr.io/core/configuration-collector:1.36.27
	event-controller ^Updated	mirantis.azurecr.io/core/event-controller:1.36.27
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.36.27
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.36.27
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.36.27
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.36.27
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.36.27
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.36.27
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.36.27
	mcc-haproxy	mirantis.azurecr.io/lcm/mcc-haproxy:v0.17.0-8-g6ca89d5
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.19.0-5-g6a7e17d
	metrics-server	mirantis.azurecr.io/core/external/metrics-server:v0.6.3-2
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.36.27
	openstack-cloud-controller-manager	mirantis.azurecr.io/lcm/kubernetes/openstack-cloud-controller-manager-amd64:v1.22.1-7-gc11024f8
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.36.27
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.36.27
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.36.27
	proxy-controller ^Updated	mirantis.azurecr.io/core/proxy-controller:1.36.27
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.36.27
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-3
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.36.27
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.36.27
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.36.27
	storage-discovery ^Deprecated	mirantis.azurecr.io/core/storage-discovery:1.36.27
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-10-g24a0d69
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.36.27
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.36.27
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.36.27

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
Helm charts	iam	http://binary.mirantis.com/iam/helm/iam-2.4.43.tgz
	iam-proxy	http://binary.mirantis.com/iam/helm/iam-proxy-0.2.16.tgz
Docker images	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.0-20200311160233
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.12-focal-20230331112513
	keycloak	mirantis.azurecr.io/iam/keycloak:0.5.16
	keycloak-gatekeeper	mirantis.azurecr.io/iam/keycloak-gatekeeper:7.1.3-4

Security notes¶

In the Container Cloud patch release 2.23.5, 70 vendor-specific Common Vulnerabilities and Exposures (CVE) have been addressed: 7 of critical and 63 of high severity.

The full list of the CVEs present in the current Container Cloud release is available at the Mirantis Security Portal.

Addressed CVEs¶
Image	Component name	CVE
bm/baremetal-dnsmasq	curl	CVE-2023-28319 (High)
		CVE-2023-28321 (High)
		CVE-2023-28322 (High)
	libcurl	CVE-2023-28319 (High)
		CVE-2023-28321 (High)
		CVE-2023-28322 (High)
	libcap2	CVE-2023-2603 (High)
	ncurses-libs	CVE-2023-29491 (High)
	ncurses-terminfo-base	CVE-2023-29491 (High)
bm/baremetal-operator	openssh-client-common	CVE-2023-28531 (Critical)
	openssh-client-default	CVE-2023-28531 (Critical)
	openssh-keygen	CVE-2023-28531 (Critical)
	ncurses-libs	CVE-2023-29491 (High)
	ncurses-terminfo-base	CVE-2023-29491 (High)
core/external/nginx	libwebp	CVE-2023-1999 (Critical)
	curl	CVE-2023-28319 (High)
		CVE-2023-28321 (High)
		CVE-2023-28322 (High)
	libcurl	CVE-2023-28319 (High)
		CVE-2023-28321 (High)
		CVE-2023-28322 (High)
core/frontend	libwebp	CVE-2023-1999 (Critical)
	curl	CVE-2023-28319 (High)
		CVE-2023-28321 (High)
		CVE-2023-28322 (High)
	libcurl	CVE-2023-28319 (High)
		CVE-2023-28321 (High)
		CVE-2023-28322 (High)
openstack/ironic	sqlparse	CVE-2023-30608 (High)
openstack/ironic-inspector	Flask	CVE-2023-30861 (High)
	sqlparse	CVE-2023-30608 (High)
stacklight/alerta-web	libcurl	CVE-2023-28319 (High)
		CVE-2023-28321 (High)
		CVE-2023-28322 (High)
	libpq	CVE-2023-2454 (High)
	postgresql15-client	CVE-2023-2454 (High)
	Flask	CVE-2023-30861 (High)
	ncurses-libs	CVE-2023-29491 (High)
	ncurses-terminfo-base	CVE-2023-29491 (High)
stacklight/alertmanager-webhook-servicenow	ncurses-libs	CVE-2023-29491 (High)
	ncurses-terminfo-base	CVE-2023-29491 (High)
stacklight/alpine-utils	curl	CVE-2023-28319 (High)
		CVE-2023-28321 (High)
		CVE-2023-28322 (High)
	libcurl	CVE-2023-28319 (High)
		CVE-2023-28321 (High)
		CVE-2023-28322 (High)
stacklight/opensearch	org.apache.santuario:xmlsec	CVE-2022-47966 (Critical)
		CVE-2022-21476 (High)
	org.slf4j:slf4j-api	CVE-2018-8088 (Critical)
	glib2	CVE-2018-16428 (High)
		CVE-2018-16429 (High)
stacklight/opensearch-dashboards	glib2	CVE-2018-16428 (High)
		CVE-2018-16429 (High)
stacklight/pgbouncer	libpq	CVE-2023-2454 (High)
	postgresql-client	CVE-2023-2454 (High)
stacklight/prometheus-libvirt-exporter	libcurl	CVE-2023-28319 (High)
		CVE-2023-28321 (High)
		CVE-2023-28322 (High)
stacklight/prometheus-patroni-exporter	ncurses-libs	CVE-2023-29491 (High)
	ncurses-terminfo-base	CVE-2023-29491 (High)
stacklight/sf-notifier	flask	CVE-2023-30861 (High)
stacklight/stacklight-toolkit	curl	CVE-2023-28319 (High)
		CVE-2023-28321 (High)
		CVE-2023-28322 (High)
	libcurl	CVE-2023-28319 (High)
		CVE-2023-28321 (High)
		CVE-2023-28322 (High)
stacklight/telegraf	github.com/docker/docker	CVE-2023-28840 (High)
		CVE-2023-28840 (High)

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.23.5 including the Cluster releases 12.7.4 and 11.7.4.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

[32761] Bare-metal nodes stuck in the cleaning state¶

During the initial deployment of Container Cloud, some nodes may get stuck in the cleaning state. As a workaround, wipe disks manually before initializing the Container Cloud bootstrap.

2.23.4¶

Container Cloud 2.23.4 is the third patch release of the 2.23.x release series that includes several addressed issues and incorporates security fixes for CVEs of Critical and High severity. This patch release:

Introduces the patch Cluster release 12.7.3 for MOSK 23.1.3.
Introduces the patch Cluster release 11.7.3
Supports the latest major Cluster releases 12.7.0, 11.7.0.
Does not support greenfield deployments based on deprecated Cluster releases 12.7.2, 11.7.2, 12.7.1, 11.7.1, 12.5.0, and 11.6.0. Use the latest available Cluster releases of the series instead.

This section describes addressed issues and contains the lists of updated artifacts and CVE fixes for the Container Cloud release 2.23.4. For CVE fixes delivered with the previous patch release, see security notes for 2.23.3 and 2.23.2.

For enhancements, addressed and known issues of the parent Container Cloud release 2.23.0, refer to 2.23.0.

Artifacts¶

This section lists the components artifacts of the Container Cloud patch release 2.23.4. For artifacts of the Cluster releases introduced in 2.23.4, see Cluster release 12.7.3 and Cluster release 11.7.3.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries	baremetal-api ^Updated	https://binary.mirantis.com/core/helm/baremetal-api-1.36.26.tgz
	baremetal-operator ^Updated	https://binary.mirantis.com/core/helm/baremetal-operator-1.36.26.tgz
	baremetal-public-api ^Updated	https://binary.mirantis.com/core/helm/baremetal-public-api-1.36.26.tgz
	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20230126190304
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20230126190304
	kaas-ipam ^Updated	https://binary.mirantis.com/core/helm/kaas-ipam-1.36.26.tgz
	local-volume-provisioner ^Updated	https://binary.mirantis.com/core/helm/local-volume-provisioner-1.36.26.tgz
	metallb ^Updated	https://binary.mirantis.com/core/helm/metallb-1.36.26.tgz
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-104-6e2e82c.tgz
Docker images	ambassador ^Updated	mirantis.azurecr.io/core/external/nginx:1.36.26
	baremetal-dnsmasq	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-alpine-20230421100738
	baremetal-operator	mirantis.azurecr.io/bm/baremetal-operator:base-focal-20230421100444
	bm-collective	mirantis.azurecr.io/bm/bm-collective:base-alpine-20230421101033
	ironic	mirantis.azurecr.io/openstack/ironic:yoga-focal-20230417060018
	ironic-inspector	mirantis.azurecr.io/openstack/ironic-inspector:yoga-focal-20230417060018
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20230330140456
	kaas-ipam	mirantis.azurecr.io/bm/kaas-ipam:base-focal-20230421100530
	mariadb ^Updated	mirantis.azurecr.io/general/mariadb:10.6.12-focal-20230423170220
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.19.0-5-g6a7e17d
	metallb-controller	mirantis.azurecr.io/bm/external/metallb/controller:v0.13.7-20221130155702-refresh-2023033102
	metallb-speaker	mirantis.azurecr.io/bm/external/metallb/speaker:v0.13.7-20221130155702-refresh-2023033102
	syslog-ng ^Updated	mirantis.azurecr.io/bm/syslog-ng:base-alpine-20230424092635

Core artifacts¶

Artifact	Component	Paths
Bootstrap tarball ^Updated	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.36.26.tar.gz
	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.36.26.tar.gz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.36.26.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.36.26.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.36.26.tgz
	byo-credentials-controller	https://binary.mirantis.com/core/helm/byo-credentials-controller-1.36.26.tgz
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.36.26.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.36.26.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.36.26.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.36.26.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.36.26.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.36.26.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.36.26.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.36.26.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.36.26.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.36.26.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.36.26.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.36.26.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.36.26.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.36.26.tgz
	metrics-server	https://binary.mirantis.com/core/helm/metrics-server-1.36.26.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.36.26.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.36.26.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.36.26.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.36.26.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.36.26.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.36.26.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.36.26.tgz
	scope-controller	http://binary.mirantis.com/core/helm/scope-controller-1.36.26.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.36.26.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.36.26.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.36.26.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.36.26.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.36.26
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.36.26
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.36.26
	byo-credentials-controller ^Updated	mirantis.azurecr.io/core/byo-credentials-controller:1.36.26
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.36.26
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.6.1
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.36.26
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.36.26
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.36.26
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.36.26
	kaas-exporter ^Updated	mirantis.azurecr.io/core/kaas-exporter:1.36.26
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.36.26
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.36.26
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.36.26
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.36.26
	mcc-haproxy	mirantis.azurecr.io/lcm/mcc-haproxy:v0.17.0-8-g6ca89d5
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.19.0-5-g6a7e17d
	metrics-server	mirantis.azurecr.io/core/external/metrics-server:v0.6.3-2
	nginx ^Updated	mirantis.azurecr.io/core/external/nginx:1.36.26
	openstack-cloud-controller-manager	mirantis.azurecr.io/lcm/kubernetes/openstack-cloud-controller-manager-amd64:v1.22.1-7-gc11024f8
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.36.26
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.36.26
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.36.26
	proxy-controller ^Updated	mirantis.azurecr.io/core/proxy-controller:1.36.26
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.36.26
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-3
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.36.26
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.36.26
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.36.26
	storage-discovery ^Deprecated	mirantis.azurecr.io/core/storage-discovery:1.36.26
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-10-g24a0d69
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.36.26
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.36.26
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.36.26

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
Helm charts	iam	http://binary.mirantis.com/iam/helm/iam-2.4.43.tgz
	iam-proxy	http://binary.mirantis.com/iam/helm/iam-proxy-0.2.16.tgz
Docker images	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.0-20200311160233
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.12-focal-20230331112513
	keycloak	mirantis.azurecr.io/iam/keycloak:0.5.16
	keycloak-gatekeeper	mirantis.azurecr.io/iam/keycloak-gatekeeper:7.1.3-4

Security notes¶

In the Container Cloud patch release 2.23.4, 35 vendor-specific CVEs have been addressed, 1 of critical and 34 of high severity.

The full list of the CVEs present in the current Container Cloud release is available at the Mirantis Security Portal.

Addressed issues¶

The following issues have been addressed in the Container Cloud patch release 2.23.4 along with the Cluster releases 12.7.3 and 11.7.3:

[31869] Fixed the issue with agent-controller failing to obtain secrets due to the incorrect indexer initialization.
[31810,30970] Fixed the issue with hardware.storage flapping in the machine status and causing constant reconciles.
[30474,28654] Fixed the issue with the agent-controller secrets leaking.
[5771] Fixed the issue with unnecessary reconciles during compute node deployment by optimizing the baremetal-provider operation.

2.23.3¶

Container Cloud 2.23.3 is the second patch release of the 2.23.x release series that incorporates security fixes for CVEs of Critical and High severity. This patch release:

Introduces the patch Cluster release 12.7.2 for MOSK 23.1.2.
Introduces the patch Cluster release 11.7.2.
Supports the latest major Cluster releases 12.7.0, 11.7.0.
Does not support greenfield deployments based on deprecated Cluster releases 12.7.1, 11.7.1, 12.5.0, and 11.6.0. Use the latest available Cluster releases of the series instead.

This section contains the lists of updated artifacts and CVE fixes for the Container Cloud release 2.23.3. For CVE fixes delivered with the previous patch release, see security notes for 2.23.2. For enhancements, addressed and known issues of the parent Container Cloud release 2.23.0, refer to 2.23.0.

Artifacts¶

This section lists the components artifacts of the Container Cloud patch release 2.23.3. For artifacts of the Cluster releases introduced in 2.23.3, see Cluster release 12.7.2 and Cluster release 11.7.2.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries	baremetal-api ^Updated	https://binary.mirantis.com/core/helm/baremetal-api-1.36.23.tgz
	baremetal-operator ^Updated	https://binary.mirantis.com/core/helm/baremetal-operator-1.36.23.tgz
	baremetal-public-api ^Updated	https://binary.mirantis.com/core/helm/baremetal-public-api-1.36.23.tgz
	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20230126190304
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20230126190304
	kaas-ipam ^Updated	https://binary.mirantis.com/core/helm/kaas-ipam-1.36.23.tgz
	local-volume-provisioner ^Updated	https://binary.mirantis.com/core/helm/local-volume-provisioner-1.36.23.tgz
	metallb ^Updated	https://binary.mirantis.com/core/helm/metallb-1.36.23.tgz
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-104-6e2e82c.tgz
Docker images	ambassador	mirantis.azurecr.io/core/external/nginx:1.36.23
	baremetal-dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-alpine-20230421100738
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-focal-20230421100444
	bm-collective ^Updated	mirantis.azurecr.io/bm/bm-collective:base-alpine-20230421101033
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:yoga-focal-20230417060018
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:yoga-focal-20230417060018
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20230330140456
	kaas-ipam ^Updated	mirantis.azurecr.io/bm/kaas-ipam:base-focal-20230421100530
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.12-focal-20230328123811
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.19.0-5-g6a7e17d
	metallb-controller	mirantis.azurecr.io/bm/external/metallb/controller:v0.13.7-20221130155702-refresh-2023033102
	metallb-speaker	mirantis.azurecr.io/bm/external/metallb/speaker:v0.13.7-20221130155702-refresh-2023033102
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-focal-20230316094816

Core artifacts¶

Artifact	Component	Paths
Bootstrap tarball ^Updated	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.36.23.tar.gz
	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.36.23.tar.gz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.36.23.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.36.23.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.36.23.tgz
	byo-credentials-controller	https://binary.mirantis.com/core/helm/byo-credentials-controller-1.36.23.tgz
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.36.23.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.36.23.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.36.23.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.36.23.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.36.23.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.36.23.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.36.23.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.36.23.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.36.23.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.36.23.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.36.23.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.36.23.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.36.23.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.36.23.tgz
	metrics-server	https://binary.mirantis.com/core/helm/metrics-server-1.36.23.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.36.23.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.36.23.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.36.23.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.36.23.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.36.23.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.36.23.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.36.23.tgz
	scope-controller	http://binary.mirantis.com/core/helm/scope-controller-1.36.23.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.36.23.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.36.23.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.36.23.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.36.23.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.36.23
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.36.23
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.36.23
	byo-credentials-controller ^Updated	mirantis.azurecr.io/core/byo-credentials-controller:1.36.23
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.36.23
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.6.1
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.36.23
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.36.23
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.36.23
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.36.23
	kaas-exporter	mirantis.azurecr.io/core/kaas-exporter:1.36.23
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.36.23
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.36.23
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.36.23
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.36.23
	mcc-haproxy	mirantis.azurecr.io/lcm/mcc-haproxy:v0.17.0-8-g6ca89d5
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.19.0-5-g6a7e17d
	metrics-server	mirantis.azurecr.io/core/external/metrics-server:v0.6.3-2
	nginx	mirantis.azurecr.io/core/external/nginx:1.36.23
	openstack-cloud-controller-manager	mirantis.azurecr.io/lcm/kubernetes/openstack-cloud-controller-manager-amd64:v1.22.1-7-gc11024f8
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.36.23
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.36.23
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.36.23
	proxy-controller ^Updated	mirantis.azurecr.io/core/proxy-controller:1.36.23
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.36.23
	registry	mirantis.azurecr.io/lcm/registry:v2.8.1-3
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.36.23
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.36.23
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.36.23
	storage-discovery ^Deprecated	mirantis.azurecr.io/core/storage-discovery:1.36.23
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-10-g24a0d69
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.36.23
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.36.23
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.36.23

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
Helm charts ^Updated	iam	http://binary.mirantis.com/iam/helm/iam-2.4.43.tgz
	iam-proxy	http://binary.mirantis.com/iam/helm/iam-proxy-0.2.16.tgz
Docker images	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.0-20200311160233
	mariadb ^Updated	mirantis.azurecr.io/general/mariadb:10.6.12-focal-20230331112513
	keycloak	mirantis.azurecr.io/iam/keycloak:0.5.16
	keycloak-gatekeeper	mirantis.azurecr.io/iam/keycloak-gatekeeper:7.1.3-4

Security notes¶

In the Container Cloud patch release 2.23.3, 28 vendor-specific CVEs have been addressed, 2 of critical and 26 of high severity.

The full list of the CVEs present in the current Container Cloud release is available at the Mirantis Security Portal.

2.23.2¶

Container Cloud 2.23.2 is the first patch release of the 2.23.x release series that incorporates security updates for CVEs with Critical and High severity. This patch release:

Introduces support for patch Cluster releases 12.7.1 and 11.7.1.
Supports the latest major Cluster releases 12.7.0 and 11.7.0.
Does not support greenfield deployments based on deprecated Cluster releases 12.5.0 and 11.6.0. Use the latest available Cluster releases of the series instead.

This section contains the lists of updated artifacts and CVE fixes for the Container Cloud release 2.23.2. For enhancements, addressed and known issues of the parent Container Cloud release 2.23.0, refer to 2.23.0.

Artifacts¶

This section lists the components artifacts of the Mirantis Container Cloud release 2.23.2. For artifacts of the Cluster releases introduced in 2.23.2, see Cluster release 12.7.1 and Cluster release 11.7.1.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶

Artifact	Component	Path
Binaries	baremetal-api ^Updated	https://binary.mirantis.com/core/helm/baremetal-api-1.36.14.tgz
	baremetal-operator ^Updated	https://binary.mirantis.com/core/helm/baremetal-operator-1.36.15.tgz
	baremetal-public-api ^Updated	https://binary.mirantis.com/core/helm/baremetal-public-api-1.36.14.tgz
	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20230126190304
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20230126190304
	kaas-ipam ^Updated	https://binary.mirantis.com/core/helm/kaas-ipam-1.36.14.tgz
	local-volume-provisioner ^Updated	https://binary.mirantis.com/core/helm/local-volume-provisioner-1.36.14.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.36.14.tgz
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-104-6e2e82c.tgz
Docker images	ambassador	mirantis.azurecr.io/core/external/nginx:1.36.14
	baremetal-dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-alpine-20230406194234
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-focal-20230405200004
	baremetal-resource-controller	n/a (merged to bm-collective)
	bm-collective ^New	mirantis.azurecr.io/bm/bm-collective:base-alpine-20230405184901
	dynamic_ipxe	n/a (merged to bm-collective)
	dnsmasq-controller	n/a (merged to bm-collective)
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:yoga-focal-20230403060017
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:yoga-focal-20230403060017
	ironic-prometheus-exporter ^Updated	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20230330140456
	kaas-ipam ^Updated	mirantis.azurecr.io/bm/kaas-ipam:base-focal-20230405184421
	mariadb ^Updated	mirantis.azurecr.io/general/mariadb:10.6.12-focal-20230328123811
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.19.0-5-g6a7e17d
	metallb-controller	mirantis.azurecr.io/bm/external/metallb/controller:v0.13.7-20221130155702-refresh-2023033102
	metallb-speaker	mirantis.azurecr.io/bm/external/metallb/speaker:v0.13.7-20221130155702-refresh-2023033102
	syslog-ng ^Updated	mirantis.azurecr.io/bm/syslog-ng:base-focal-20230316094816

Core artifacts¶

Artifact	Component	Paths
Bootstrap tarball ^Updated	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.36.14.tar.gz
	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.36.14.tar.gz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.36.14.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.36.14.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.36.14.tgz
	byo-credentials-controller	https://binary.mirantis.com/core/helm/byo-credentials-controller-1.36.14.tgz
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.36.14.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.36.14.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.36.14.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.36.14.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.36.14.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.36.14.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.36.14.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.36.14.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.36.14.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.36.14.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.36.14.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.36.14.tgz
	machinepool-controller	https://binary.mirantis.com/core/helm/machinepool-controller-1.36.14.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.36.14.tgz
	metrics-server	https://binary.mirantis.com/core/helm/metrics-server-1.36.14.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.36.14.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.36.14.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.36.14.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.36.14.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.36.14.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.36.14.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.36.14.tgz
	scope-controller	http://binary.mirantis.com/core/helm/scope-controller-1.36.14.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.36.14.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.36.14.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.36.14.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.36.14.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.36.14
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.36.14
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.36.14
	byo-credentials-controller ^Updated	mirantis.azurecr.io/core/byo-credentials-controller:1.36.14
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.36.14
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.6.1
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.36.14
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.36.14
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.36.14
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.36.14
	kaas-exporter	mirantis.azurecr.io/core/kaas-exporter:1.36.14
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.36.14
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.36.14
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.36.14
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.36.14
	mcc-haproxy	mirantis.azurecr.io/lcm/mcc-haproxy:v0.17.0-8-g6ca89d5
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.19.0-5-g6a7e17d
	metrics-server	mirantis.azurecr.io/core/external/metrics-server:v0.6.3-2
	nginx	mirantis.azurecr.io/core/external/nginx:1.36.14
	openstack-cloud-controller-manager	mirantis.azurecr.io/lcm/kubernetes/openstack-cloud-controller-manager-amd64:v1.22.1-7-gc11024f8
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.36.14
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.36.14
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.36.14
	proxy-controller ^Updated	mirantis.azurecr.io/core/proxy-controller:1.36.14
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.36.14
	registry ^Updated	mirantis.azurecr.io/lcm/registry:v2.8.1-1-g7bde01d2
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.36.14
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.36.14
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.36.14
	squid-proxy ^Updated	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-10-g24a0d69
	storage-discovery ^Deprecated	mirantis.azurecr.io/core/storage-discovery:1.36.14
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.36.14
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.36.14
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.36.14

IAM artifacts¶

Artifact	Component	Path
Binaries	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
Helm charts ^Updated	iam	http://binary.mirantis.com/iam/helm/iam-2.4.41.tgz
	iam-proxy	http://binary.mirantis.com/iam/helm/iam-proxy-0.2.16.tgz
Docker images	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.0-20200311160233
	mariadb ^Updated	mirantis.azurecr.io/general/mariadb:10.6.12-focal-20230227122722
	keycloak ^Updated	mirantis.azurecr.io/iam/keycloak:0.5.16
	keycloak-gatekeeper ^Updated	mirantis.azurecr.io/iam/keycloak-gatekeeper:7.1.3-4

Security notes¶

In Container Cloud 2.23.2, 1087 vendor-specific CVEs have been addressed, 53 with critical and 1034 with high severity.

The full list of the CVEs present in the current Container Cloud release is available at the Mirantis Security Portal.

2.23.1¶

The Mirantis Container Cloud GA release 2.23.1 is based on 2.23.0 and:

Introduces support for the Cluster release 12.7.0 that is based on the Cluster release 11.7.0 and represents Mirantis OpenStack for Kubernetes (MOSK) 23.1.

This Cluster release is based on the updated version of Mirantis Kubernetes Engine 3.5.7 with Kubernetes 1.21 and Mirantis Container Runtime 20.10.13.
Supports the latest Cluster release 11.7.0
Does not support greenfield deployments based on deprecated Cluster releases 12.5.0 and 11.6.0. Use the latest available Cluster releases of the series instead.

For details about the Container Cloud release 2.23.1, refer to its parent releases 2.23.0 and 2.22.0:

Enhancements for 2.23.0 and 2.22.0
Addressed issues for 2.23.0 and 2.22.0
Known issues
Components versions
Artifacts

Caution

See also

2.23.0¶

The Mirantis Container Cloud GA release 2.23.0:

Introduces support for the Cluster release 11.7.0 that is based on Mirantis Container Runtime 20.10.13 and Mirantis Kubernetes Engine 3.5.7 with Kubernetes 1.21.
Supports the Cluster release 12.5.0 that is based on the Cluster release 11.5.0 and represents Mirantis OpenStack for Kubernetes (MOSK) 22.5.
Does not support greenfield deployments on deprecated Cluster releases 11.6.0, 8.10.0, and 7.11.0. Use the latest available Cluster releases of the series instead.

Caution

Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.23.0.

Enhancements¶

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.23.0. For the list of enhancements in the Cluster release 11.7.0 that is introduced by the Container Cloud release 2.23.0, see the Cluster releases (managed).

Graceful cluster reboot
Readiness fields for ‘Machine’ and ‘Cluster’ objects
Deletion of persistent volumes during an OpenStack-based cluster deletion
Ability to disable time sync management
The ‘Upgrade’ button for easy cluster update through the web UI

Graceful cluster reboot¶

Implemented the capability to perform a graceful reboot on a management, regional, or managed cluster for all supported providers using the GracefulRebootRequest custom resource. Use this resource for a rolling reboot of several or all cluster machines without workloads interruption. The reboot occurs in the order of cluster upgrade policy.

The resource is also useful for a bulk reboot of machines, for example, on large clusters.

To verify the reboot status of a machine:

kubectl get machines  <machineName> -o wide

Example of system response:

NAME    READY  LCMPHASE  NODENAME            UPGRADEINDEX  REBOOTREQUIRED  WARNINGS
demo-0  true   Ready     kaas-node-c6aa8ad3  1             true

Note

For MOSK-based deployments, the feature support is available since MOSK 23.1.

Learn more

Readiness fields for ‘Machine’ and ‘Cluster’ objects¶

Enhanced Machine and Cluster objects by adding the following output columns to the kubectl get machines -o wide and kubectl get cluster -o wide commands to simplify monitoring of machine and cluster states. More specifically, you can now obtain the following machine and cluster details:

Machine object:
- READY
- UPGRADEINDEX
- REBOOTREQUIRED
- WARNINGS
- LCMPHASE (renamed from PHASE)
Cluster object:
- READY
- RELEASE
- WARNINGS

Example system response of the kubectl get machines <machineName> -o wide command:

NAME    READY  LCMPHASE  NODENAME            UPGRADEINDEX  REBOOTREQUIRED  WARNINGS
demo-0  true   Ready     kaas-node-c6aa8ad3  1             true

Learn more

Operations Guide: Create and operate managed clusters

Deletion of persistent volumes during an OpenStack-based cluster deletion¶

TechPreview

Implemented the initial Technology Preview API support for deletion of persistent volumes during an OpenStack-based managed cluster deletion. To enable the feature, set the boolean volumesCleanupEnabled option in the spec.providerSpec.value section of the Cluster object before a managed cluster deletion.

Ability to disable time sync management¶

Implemented the capability to disable time sync management during a management or regional cluster bootstrap using the ntpEnabled=false option. The default setting remains ntpEnabled=true. The feature disables the management of chrony configuration by Container Cloud and enables you to use your own system for chrony management.

Note

For MOSK-based deployments, the feature support is available since MOSK 23.1.

Learn more

Deployment Guide: Deploy a management or regional cluster

The ‘Upgrade’ button for easy cluster update through the web UI¶

Implemented a separate Upgrade button in the Container Cloud web UI to simplify the start of a cluster update. This button provides easy access to the cluster update dialog and has the same functionality as the Upgrade cluster option available under the cluster menu.

The Upgrade button is located on the Clusters page next to the More action icon located in the last column for each cluster when a new Cluster release update becomes available.

If the Upgrade button is greyed out, the cluster is in maintenance mode that must be disabled before you can proceed with cluster update. For details, see Enable maintenance mode on a cluster and machine using web UI.

If the Upgrade button does not display, your cluster is up-to-date.

Learn more

Operations Guide: Update a managed cluster

Addressed issues¶

The following issues have been addressed in the Mirantis Container Cloud release 2.23.0 along with the Cluster release 11.7.0:

[29647] Fixed the issue with the Network prepared stage getting stuck in the NotStarted status during deployment of a vSphere-based management or regional cluster with IPAM disabled.
[26896] Fixed the issue with the MetalLB liveness and readiness timeouts in a slow network.
[28313] Fixed the issue with the iam-keycloak Pod starting slowly because of DB errors causing timeouts while waiting for the OIDC configuration readiness.
[28675] Fixed the issue with the Ceph OSD-related parameters configured using rookConfig in KaaSCephcluster being not applied until OSDs are restarted. Now, parameters for Ceph OSD daemons apply during runtime instead of setting them directly in ceph.conf. Therefore, no restart is required.
[30040] Fixed the issue with the HelmBundleReleaseNotDeployed alert that has the release_name=opensearch label firing during the Container Cloud or Cluster release update due to issues with the claim request size in the elasticsearch.persistentVolumeClaimSize configuration.
[29329] Fixed the issue with recreation of the Patroni container replica being stuck in the degraded state due to the liveness probe killing the container that runs the pg_rewind procedure during cluster update.
[28822] Fixed the issue with Reference Application triggering false-positive alerts related to Reference Application during its upgrade.
[28479] Fixed the issue with the restarts count of the metric-collector Pod being increased in time with reason: OOMKilled in containerStatuses of the metric-collector Pod on baremetal-based management clusters with HTTP proxy enabled.
[28417] Fixed the issue with the Reports Dashboards plugin not being enabled by default preventing the use of the reporting option. For details about this plugin, see the GitHub OpenSearch documentation: OpenSearch Dashboards Reports.
[28373] Fixed the issue with Alerta getting stuck after a failed initialization during cluster creation with StackLight enabled.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.23.0 including the Cluster release 11.7.0.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal
LCM
TLS configuration
Ceph
StackLight

Bare metal¶

[29762] Wrong IP address is assigned after the MetalLB controller restart¶

Fixed in 14.0.0(1) and 15.0.1

Due to the upstream MetalLB issue, a race condition occurs when assigning an IP address after the MetalLB controller restart. If a new service of the LoadBalancer type is created during the MetalLB Controller restart, then this service can be assigned an IP address that was already assigned to another service before the MetalLB Controller restart.

To verify that the cluster is affected:

Verify whether IP addresses of the LoadBalancer (LB) type are duplicated where they are not supposed to:

kubectl get svc -A|grep LoadBalancer

Note

Some services use shared IP addresses on purpose. In the example system response below, these are services using the IP address 10.0.1.141.

Example system response:

kaas        dhcp-lb                   LoadBalancer  10.233.4.192   10.0.1.141      53:32594/UDP,67:30048/UDP,68:30464/UDP,69:31898/UDP,123:32450/UDP  13h
kaas        dhcp-lb-tcp               LoadBalancer  10.233.6.79    10.0.1.141      8080:31796/TCP,53:32012/TCP                                        11h
kaas        httpd-http                LoadBalancer  10.233.0.92    10.0.1.141      80:30115/TCP                                                       13h
kaas        iam-keycloak-http         LoadBalancer  10.233.55.2    10.100.91.101   443:30858/TCP,9990:32301/TCP                                       2h
kaas        ironic-kaas-bm            LoadBalancer  10.233.26.176  10.0.1.141      6385:31748/TCP,8089:30604/TCP,5050:32200/TCP,9797:31988/TCP,601:31888/TCP 13h
kaas        ironic-syslog             LoadBalancer  10.233.59.199  10.0.1.141      514:32098/UDP                                                      13h
kaas        kaas-kaas-ui              LoadBalancer  10.233.51.167  10.100.91.101   443:30976/TCP                                                      13h
kaas        mcc-cache                 LoadBalancer  10.233.40.68   10.100.91.102   80:32278/TCP,443:32462/TCP                                         12h
kaas        mcc-cache-pxe             LoadBalancer  10.233.10.75   10.0.1.142      80:30112/TCP,443:31559/TCP                                         12h
stacklight  iam-proxy-alerta          LoadBalancer  10.233.4.102   10.100.91.104   443:30101/TCP                                                      12h
stacklight  iam-proxy-alertmanager    LoadBalancer  10.233.46.45   10.100.91.105   443:30944/TCP                                                      12h
stacklight  iam-proxy-grafana         LoadBalancer  10.233.39.24   10.100.91.106   443:30953/TCP                                                      12h
stacklight  iam-proxy-prometheus      LoadBalancer  10.233.12.174  10.100.91.107   443:31300/TCP                                                      12h
stacklight  telemeter-server-external LoadBalancer  10.233.56.63   10.100.91.103   443:30582/TCP                                                      12h

In the above example, the iam-keycloak-http and kaas-kaas-ui services erroneously use the same IP address 10.100.91.101. They both use the same port 443 producing a collision when an application tries to access the 10.100.91.101:443 endpoint.

Workaround:

Unassign the current LB IP address for the selected service, as no LB IP address can be used for the NodePort service:
```
kubectl -n kaas patch svc <serviceName> -p '{"spec":{"type":"NodePort"}}'
```
Assign a new LB IP address for the selected service:
```
kubectl -n kaas patch svc <serviceName> -p '{"spec":{"type":"LoadBalancer"}}'
```
The second affected service will continue using its current LB IP address.

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

[20736] Region deletion failure after regional deployment failure¶

If a baremetal-based regional cluster deployment fails before pivoting is done, the corresponding region deletion fails.

Workaround:

cluster
machine
baremetalhost
baremetalhostprofile
l2template
subnet
ipamhost
ipaddr

kubectl delete <objectName> -l kaas.mirantis.com/region=<regionName>

Warning

Do not use the same region name again after the regional cluster deployment failure since some objects that reference the region name may still exist.

LCM¶

[5981] Upgrade gets stuck on the cluster with more that 120 nodes¶

Fixed in 14.0.0(1) and 15.0.1

Upgrade of a cluster with more than 120 nodes gets stuck with errors about IP addresses exhaustion in the docker logs.

Note

If you plan to scale your cluster to more than 120 nodes, the cluster will be affected by the issue. Therefore, you will have to perform the workaround below.

Workaround:

Caution

If you have not run the cluster upgrade yet, simply recreate the mke-overlay network as described in the step 6 and skip all other steps.

Note

If you successfully upgraded the cluster with less than 120 nodes but plan to scale it to more that 120 node, proceed with steps 2-9.

Verify that MKE nodes are upgraded:

On any master node, run the following command to identify ucp-worker-agent that has a newer version:

docker service ls

Example of system response:

ID             NAME                     MODE         REPLICAS   IMAGE                          PORTS
7jdl9m0giuso   ucp-3-5-7                global       0/0        mirantis/ucp:3.5.7
uloi2ixrd0br   ucp-auth-api             global       3/3        mirantis/ucp-auth:3.5.7
pfub4xa17nkb   ucp-auth-worker          global       3/3        mirantis/ucp-auth:3.5.7
00w1kqn0x69w   ucp-cluster-agent        replicated   1/1        mirantis/ucp-agent:3.5.7
xjhwv1vrw9k5   ucp-kube-proxy-win       global       0/0        mirantis/ucp-agent-win:3.5.7
oz28q8a7swmo   ucp-kubelet-win          global       0/0        mirantis/ucp-agent-win:3.5.7
ssjwonmnvk3s   ucp-manager-agent        global       3/3        mirantis/ucp-agent:3.5.7
ks0ttzydkxmh   ucp-pod-cleaner-win      global       0/0        mirantis/ucp-agent-win:3.5.7
w5d25qgneibv   ucp-tigera-felix-win     global       0/0        mirantis/ucp-agent-win:3.5.7
ni86z33o10n3   ucp-tigera-node-win      global       0/0        mirantis/ucp-agent-win:3.5.7
iyyh1f0z6ejc   ucp-worker-agent-win-x   global       0/0        mirantis/ucp-agent-win:3.5.5
5z6ew4fmf2mm   ucp-worker-agent-win-y   global       0/0        mirantis/ucp-agent-win:3.5.7
gr52h05hcwwn   ucp-worker-agent-x       global       56/56      mirantis/ucp-agent:3.5.5
e8coi9bx2j7j   ucp-worker-agent-y       global       121/121    mirantis/ucp-agent:3.5.7

In the above example, it is ucp-worker-agent-y.

Obtain the node list:

docker service ps ucp-worker-agent-y | awk -F ' ' ‘$4 ~ /^kaas/ {print $4}’ > upgraded_nodes.txt

Identify the cluster ID. For example, run the following command on the management cluster:

kubectl -n <clusterNamespace> get cluster <clusterName> -o json | jq '.status.providerStatus.mke.clusterID'

Create a backup of MKE as described in the MKE documentation: Backup procedure.

Remove MKE services:

docker service rm ucp-cluster-agent ucp-manager-agent ucp-worker-agent-win-y ucp-worker-agent-y ucp-worker-agent-win-x ucp-worker-agent-x

Remove the mke-overlay network:
```
docker network rm mke-overlay
```
Recreate the mke-overlay network with a correct CIDR that must be at least /20 and have no interventions with other subnets in the cluster network. For example:
```
docker network create -d overlay --subnet 10.1.0.0/20 mke-overlay
```

Create placeholder worker services:

docker service create --name ucp-worker-agent-x --mode global --constraint node.labels.foo==bar --detach busybox sleep 3d

docker service create --name ucp-worker-agent-win-x --mode global --constraint node.labels.foo==bar --detach busybox sleep 3d

Recreate all MKE services using the previously obtained cluster ID. Use the target version for your cluster, for example, 3.5.7:

docker container run --rm -it --name ucp -v /var/run/docker.sock:/var/run/docker.sock mirantis/ucp:3.5.7 upgrade --debug --manual-worker-upgrade --force-minimums --id <cluster ID> --interactive --force-port-check

Note

Because of interactive mode, you may need to use Ctrl+C when the command execution completes.

Verify that all services are recreated:
```
docker service ls
```
The exemplary ucp-worker-agent-y service must have 1 replica running with a node that was previously stuck.
Using the node list obtained in the first step, remove the upgrade-hold labels from the nodes that were previously upgraded:
```
for i in $(cat upgraded_nodes.txt); do docker node update --label-rm com.docker.ucp.upgrade-hold $i; done
```
Verify that all nodes from the list obtained in the first step are present in the ucp-worker-agent-y service. For example:
```
docker service ps ucp-worker-agent-y
```

[5782] Manager machine fails to be deployed during node replacement¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a manager machine, the following problems may occur:

The system adds the node to Docker swarm but not to Kubernetes
The node Deployment gets stuck with failed RethinkDB health checks

Workaround:

Delete the failed node.
Wait for the MKE cluster to become healthy. To monitor the cluster status:
1. Log in to the MKE web UI as described in Connect to the Mirantis Kubernetes Engine web UI.
2. Monitor the cluster status as described in MKE Operations Guide: Monitor an MKE cluster with the MKE web UI.
Deploy a new node.

[5568] The calico-kube-controllers Pod fails to clean up resources¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During the unsafe or forced deletion of a manager machine running the calico-kube-controllers Pod in the kube-system namespace, the following issues occur:

The calico-kube-controllers Pod fails to clean up resources associated with the deleted node
The calico-node Pod may fail to start up on a newly created node if the machine is provisioned with the same IP address as the deleted machine had

As a workaround, before deletion of the node running the calico-kube-controllers Pod, cordon and drain the node:

kubectl cordon <nodeName>
kubectl drain <nodeName>

[30294] Replacement of a master node is stuck on the calico-node Pod start¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a master node on a cluster of any type, the calico-node Pod fails to start on a new node that has the same IP address as the node being replaced.

Workaround:

From a CLI with an MKE client bundle, create a shell alias to start calicoctl using the mirantis/ucp-dsinfo image:

Since MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/key.pem \
-e ETCD_CA_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/ca.pem \
-e ETCD_CERT_FILE=/var/lib/docker/volumes/ucp-kv-certs/_data/cert.pem \
-v /var/run/calico:/var/run/calico \
-v /var/lib/docker/volumes/ucp-kv-certs/_data:/var/lib/docker/volumes/ucp-kv-certs/_data:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl \
"

Before MKE 3.7.2

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=<etcdEndpoint> \
-e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
-e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
-e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
-v /var/run/calico:/var/run/calico \
-v ucp-node-certs:/ucp-node-certs:ro \
mirantis/ucp-dsinfo:<mkeVersion> \
calicoctl --allow-version-mismatch \
"

In the above command, replace the following values with the corresponding settings of the affected cluster:

<etcdEndpoint> is the etcd endpoint defined in the Calico configuration file. For example, ETCD_ENDPOINTS=127.0.0.1:12378
<mkeVersion> is the MKE version installed on your cluster. For example, mirantis/ucp-dsinfo:3.5.7.

Verify the node list on the cluster:
```
kubectl get node
```
Compare this list with the node list in Calico to identify the old node:
```
calicoctl get node -o wide
```

Remove the old node from Calico:

calicoctl delete node kaas-node-<nodeID>

[27797] A cluster ‘kubeconfig’ stops working during MKE minor version update¶

During update of a Container Cloud cluster of any type, if the MKE minor version is updated from 3.4.x to 3.5.x, access to the cluster using the existing kubeconfig fails with the You must be logged in to the server (Unauthorized) error due to OIDC settings being reconfigured.

As a workaround, during the cluster update process, use the admin kubeconfig instead of the existing one. Once the update completes, you can use the existing cluster kubeconfig again.

To obtain the admin kubeconfig:

kubectl --kubeconfig <pathToMgmtKubeconfig> get secret -n <affectedClusterNamespace> \
-o yaml <affectedClusterName>-kubeconfig | awk '/admin.conf/ {print $2}' | \
head -1 | base64 -d > clusterKubeconfig.yaml

If the related cluster is regional, replace <pathToMgmtKubeconfig> with <pathToRegionalKubeconfig>.

TLS configuration¶

[29604] The ‘failed to get kubeconfig’ error during TLS configuration¶

Fixed in 14.0.0(1) and 15.0.1

When setting a new Transport Layer Security (TLS) certificate for a cluster, the false positive failed to get kubeconfig error may occur on the Waiting for TLS settings to be applied stage. No actions are required. Therefore, disregard the error.

To verify the status of the TLS configuration being applied:

kubectl get cluster <ClusterName> -n <ClusterProjectName> -o jsonpath-as-json="{.status.providerStatus.tls.<Application>}"

Possible values for the <Application> parameter are as follows:

keycloak
ui
cache
mke
iamProxyAlerta
iamProxyAlertManager
iamProxyGrafana
iamProxyKibana
iamProxyPrometheus

Example of system response:

[
    {
        "expirationTime": "2024-01-06T09:37:04Z",
        "hostname": "domain.com",
    }
]

In this example, expirationTime equals the NotAfter field of the server certificate. And the value of hostname contains the configured application name.

Ceph¶

[30857] Irrelevant error during Ceph OSD deployment on removable devices¶

Fixed in 14.0.0(1) and 15.0.1

The deployment of Ceph OSDs fails with the following messages in the status section of the KaaSCephCluster custom resource:

shortClusterInfo:
  messages:
  - Not all osds are deployed
  - Not all osds are in
  - Not all osds are up

To find out if your cluster is affected, verify if the devices on the AMD hosts you use for the Ceph OSDs deployment are removable. For example, if the sdb device name is specified in spec.cephClusterSpec.nodes.storageDevices of the KaaSCephCluster custom resource for the affected host, run:

# cat /sys/block/sdb/removable
1

The system output shows that the reason of the above messages in status is the enabled hotplug functionality on the AMD nodes, which marks all drives as removable. And the hotplug functionality is not supported by Ceph in Container Cloud.

As a workaround, disable the hotplug functionality in the BIOS settings for disks that are configured to be used as Ceph OSD data devices.

[30635] Ceph ‘pg_autoscaler’ is stuck with the ‘overlapping roots’ error¶

Fixed in 14.0.0(1) and 15.0.1

Due to the upstream Ceph issue occurring since Ceph Pacific, the pg_autoscaler module of Ceph Manager fails with the pool <poolNumber> has overlapping roots error if a Ceph cluster contains a mix of pools with deviceClass either explicitly specified or not specified.

The deviceClass parameter is required for a pool definition in the spec section of the KaaSCephCluster object, but not required for Ceph RADOS Gateway (RGW) and Ceph File System (CephFS). Therefore, if sections for Ceph RGW or CephFS data or metadata pools are defined without deviceClass, then autoscaling of placement groups is disabled on a cluster due to overlapping roots. Overlapping roots imply that Ceph RGW and/or CephFS pools obtained the default crush rule and have no demarcation on a specific class to store data.

Note

If pools for Ceph RGW and CephFS already have deviceClass specified, skip the corresponding steps of the below procedure.

Note

Perform the below procedure on the affected managed cluster using its kubeconfig.

Workaround:

Obtain failureDomain and required replicas for Ceph RGW and/or CephFS pools:

Note

If the KaasCephCluster spec section does not contain failureDomain, failureDomain equals host by default to store one replica per node.
Note

The types of pools crush rules include:
- An erasureCoded pool requires the codingChunks + dataChunks number of available units of failureDomain.
- A replicated pool requires the replicated.size number of available units of failureDomain.
- To obtain Ceph RGW pools, use the spec.cephClusterSpec.objectStorage.rgw section of the KaaSCephCluster object. For example:
```
objectStorage:
  rgw:
    dataPool:
      failureDomain: host
      erasureCoded:
        codingChunks: 1
        dataChunks: 2
    metadataPool:
      failureDomain: host
      replicated:
        size: 3
    gateway:
      allNodes: false
      instances: 3
      port: 80
      securePort: 8443
    name: openstack-store
    preservePoolsOnDelete: false
```
  The dataPool pool requires the sum of codingChunks and dataChunks values representing the number of available units of failureDomain. In the example above, for failureDomain: host, dataPool requires 3 available nodes to store its objects.
  
  The metadataPool pool requires the replicated.size number of available units of failureDomain. For failureDomain: host, metadataPool requires 3 available nodes to store its objects.
- To obtain CephFS pools, use the spec.cephClusterSpec.sharedFilesystem.cephFS section of the KaaSCephCluster object. For example:
```
sharedFilesystem:
  cephFS:
  - name: cephfs-store
    dataPools:
    - name: default-pool
      replicated:
        size: 3
      failureDomain: host
    - name: second-pool
      erasureCoded:
        dataChunks: 2
        codingChunks: 1
    metadataPool:
      replicated:
        size: 3
      failureDomain: host
    ...
```
  The default-pool and metadataPool pools require the replicated.size number of available units of failureDomain. For failureDomain: host, default-pool requires 3 available nodes to store its objects.
  
  The second-pool pool requires the sum of codingChunks and dataChunks representing the number of available units of failureDomain. For failureDomain: host, second-pool requires 3 available nodes to store its objects.

Obtain the device class that meets the desired number of required replicas for the defined failureDomain.

Calculate potential data size for Ceph RGW and CephFS pools.

Calculation of data size

Obtain Ceph data stored by classes and pools:

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph df

Example output:

--- RAW STORAGE ---
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
hdd     96 GiB   90 GiB  6.0 GiB   6.0 GiB       6.26
ssd     96 GiB   96 GiB  211 MiB   211 MiB       0.21
TOTAL  192 GiB  186 GiB  6.2 GiB   6.2 GiB       3.24

--- POOLS ---
POOL                                ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
device_health_metrics                1    1      0 B        0      0 B      0     42 GiB
kubernetes-hdd                       2   32  2.3 GiB      707  4.6 GiB   5.15     42 GiB
kubernetes-2-ssd                    11   32     19 B        1    8 KiB      0     45 GiB
openstack-store.rgw.meta            12   32  2.5 KiB       10   64 KiB      0     45 GiB
openstack-store.rgw.log             13   32   23 KiB      309  1.3 MiB      0     45 GiB
.rgw.root                           14   32  4.8 KiB       16  120 KiB      0     45 GiB
openstack-store.rgw.otp             15   32      0 B        0      0 B      0     45 GiB
openstack-store.rgw.control         16   32      0 B        8      0 B      0     45 GiB
openstack-store.rgw.buckets.index   17   32  2.7 KiB       22  5.3 KiB      0     45 GiB
openstack-store.rgw.buckets.non-ec  18   32      0 B        0      0 B      0     45 GiB
openstack-store.rgw.buckets.data    19   32  103 MiB       26  155 MiB   0.17     61 GiB

Summarize the USED size of all <rgwName>.rgw.* pools and compare it with the AVAIL size of each applicable device class selected in the previous step.

Note

As Ceph RGW pools lack explicit specification of deviceClass, they may store objects on all device classes. The resulted device size can be smaller than the calculated USED size because part of data can already be stored in the desired class. Therefore, limiting pools to a single device class may result in a smaller occupied data size than the total USED size. Nonetheless, calculating the USED size of all pools remains valid because the pool data may not be stored on the selected device class.
For CephFS data or metadata pools, use the previous step to calculate the USED size of pools and compare it with the AVAIL size.
Decide which device class from applicable by required replicas and available size is more preferable to store Ceph RGW and CephFS data. In the example output above, hdd and ssd are both applicable. Therefore, select any of them.

Note

You can select different device classes for Ceph RGW and CephFS. For example, hdd for Ceph RGW and ssd for CephFS. Select a device class based on performance expectations, if any.

Create the rule-helper script to switch Ceph RGW or CephFS pools to a device usage.

Creation of the rule-helper script

Create the rule-helper script file:

Get a shell of the ceph-tools Pod:

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash

Create the /tmp/rule-helper.py file with the following content:

cat > /tmp/rule-helper.py << EOF
import argparse
import json
import subprocess
from sys import argv, exit


def get_cmd(cmd_args):
    output_args = ['--format', 'json']
    _cmd = subprocess.Popen(cmd_args + output_args,
                            stdout=subprocess.PIPE,
                            stderr=subprocess.PIPE)
    stdout, stderr = _cmd.communicate()
    if stderr:
        error = stderr
        print("[ERROR] Failed to get '{0}': {1}".format(cmd_args.join(' '), stderr))
        return
    return stdout


def format_step(action, cmd_args):
    return "{0}:\n\t{1}".format(action, ' '.join(cmd_args))


def process_rule(rule):
    steps = []
    new_rule_name = rule['rule_name'] + '_v2'
    if rule['type'] == "replicated":
        rule_create_args = ['ceph', 'osd', 'crush', 'create-replicated',
            new_rule_name, rule['root'], rule['failure_domain'], rule['device_class']]
        steps.append(format_step("create a new replicated rule for pool", rule_create_args))
    else:
        new_profile_name = rule['profile_name'] + '_' + rule['device_class']
        profile_create_args = ['ceph', 'osd', 'erasure-code-profile', 'set', new_profile_name]
        for k,v in rule['profile'].items():
            profile_create_args.append("{0}={1}".format(k,v))
        rule_create_args = ['ceph', 'osd', 'crush', 'create-erasure', new_rule_name, new_profile_name]
        steps.append(format_step("create a new erasure-coded profile", profile_create_args))
        steps.append(format_step("create a new erasure-coded rule for pool", rule_create_args))

    set_rule_args = ['ceph', 'osd', 'pool', 'set', 'crush_rule', rule['pool_name'], new_rule_name]
    revert_rule_args = ['ceph', 'osd', 'pool', 'set', 'crush_rule', new_rule_name, rule['pool_name']]
    rm_old_rule_args = ['ceph', 'osd', 'crush', 'rule', 'rm', rule['rule_name']]
    rename_rule_args = ['ceph', 'osd', 'crush', 'rule', 'rename', new_rule_name, rule['rule_name']]
    steps.append(format_step("set pool crush rule to new one", set_rule_args))
    steps.append("check that replication is finished and status healthy: ceph -s")
    steps.append(format_step("in case of any problems revert step 2 and stop procedure", revert_rule_args))
    steps.append(format_step("remove standard (old) pool crush rule", rm_old_rule_args))
    steps.append(format_step("rename new pool crush rule to standard name", rename_rule_args))
    if rule['type'] != "replicated":
        rm_old_profile_args = ['ceph', 'osd', 'erasure-code-profile', 'rm', rule['profile_name']]
        steps.append(format_step("remove standard (old) erasure-coded profile", rm_old_profile_args))

    for idx, step in enumerate(steps):
        print("  {0}) {1}".format(idx+1, step))


def check_rules(args):
    extra_pools_lookup = []
    if args.type == "rgw":
        extra_pools_lookup.append(".rgw.root")
    pools_str = get_cmd(['ceph', 'osd', 'pool', 'ls', 'detail'])
    if pools_str == '':
        return
    rules_str = get_cmd(['ceph', 'osd', 'crush', 'rule', 'dump'])
    if rules_str == '':
        return
    try:
        pools_dump = json.loads(pools_str)
        rules_dump = json.loads(rules_str)
        if len(pools_dump) == 0:
            print("[ERROR] No pools found")
            return
        if len(rules_dump) == 0:
            print("[ERROR] No crush rules found")
            return
        crush_rules_recreate = []
        for pool in pools_dump:
            if pool['pool_name'].startswith(args.prefix) or pool['pool_name'] in extra_pools_lookup:
                rule_id = pool['crush_rule']
                for rule in rules_dump:
                    if rule['rule_id'] == rule_id:
                        recreate = False
                        new_rule = {'rule_name': rule['rule_name'], 'pool_name': pool['pool_name']}
                        for step in rule.get('steps',[]):
                            root = step.get('item_name', '').split('~')
                            if root[0] != '' and len(root) == 1:
                                new_rule['root'] = root[0]
                                continue
                            failure_domain = step.get('type', '')
                            if failure_domain != '':
                                new_rule['failure_domain'] = failure_domain
                        if new_rule.get('root', '') == '':
                            continue
                        new_rule['device_class'] = args.device_class
                        if pool['erasure_code_profile'] == "":
                            new_rule['type'] = "replicated"
                        else:
                            new_rule['type'] = "erasure"
                            profile_str = get_cmd(['ceph', 'osd', 'erasure-code-profile', 'get', pool['erasure_code_profile']])
                            if profile_str == '':
                                return
                            profile_dump = json.loads(profile_str)
                            profile_dump['crush-device-class'] = args.device_class
                            new_rule['profile_name'] = pool['erasure_code_profile']
                            new_rule['profile'] = profile_dump
                        crush_rules_recreate.append(new_rule)
                        break
        print("Found {0} pools with crush rules require device class set".format(len(crush_rules_recreate)))
        for new_rule in crush_rules_recreate:
            print("- Pool {0} requires crush rule update, device class is not set".format(new_rule['pool_name']))
            process_rule(new_rule)
    except Exception as err:
        print("[ERROR] Failed to get info from Ceph: {0}".format(err))
        return


if __name__ == '__main__':
    parser = argparse.ArgumentParser(
        description='Ceph crush rules checker. Specify device class and service name.',
        prog=argv[0], usage='%(prog)s [options]')
    parser.add_argument('--type', type=str,
                        help='Type of pool: rgw, cephfs',
                        default='',
                        required=True)
    parser.add_argument('--prefix', type=str,
                        help='Pool prefix. If objectstore - use objectstore name, if CephFS - CephFS name.',
                        default='',
                        required=True)
    parser.add_argument('--device-class', type=str,
                        help='Device class to switch on.',
                        required=True)
    args = parser.parse_args()
    if len(argv) < 3:
        parser.print_help()
        exit(0)

    check_rules(args)
EOF

Exit the ceph-tools Pod.

For Ceph RGW, execute the rule-helper script to output the step-by-step instruction and run each step provided in the output manually.

Note

The following steps include creation of crush rules with the same parameters as before but with the device class specification and switching of pools to new crush rules.

Execution of the rule-helper script steps for Ceph RGW

Get a shell of the ceph-tools Pod:

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash

Run the /tmp/rule-helper.py script with the following parameters:
```
python3 /tmp/rule-helper.py --prefix <rgwName> --type rgw --device-class <deviceClass>
```
Substitute the following parameters:
- <rgwName> with the Ceph RGW name from spec.cephClusterSpec.objectStorage.rgw.name in the KaaSCephCluster object. In the example above, the name is openstack-store.
- <deviceClass> with the device class selected in the previous steps.

Using the output of the command from the previous step, run manual commands step-by-step.

Example output for the hdd device class:

Found 7 pools with crush rules require device class set
- Pool openstack-store.rgw.control requires crush rule update, device class is not set
  1) create a new replicated rule for pool:
    ceph osd crush create-replicated openstack-store.rgw.control_v2 default host hdd
  2) set pool crush rule to new one:
    ceph osd pool set crush_rule openstack-store.rgw.control openstack-store.rgw.control_v2
  3) check that replication is finished and status healthy: ceph -s
  4) in case of any problems revert step 2 and stop procedure:
    ceph osd pool set crush_rule openstack-store.rgw.control_v2 openstack-store.rgw.control
  5) remove standard (old) pool crush rule:
    ceph osd crush rule rm openstack-store.rgw.control
  6) rename new pool crush rule to standard name:
    ceph osd crush rule rename openstack-store.rgw.control_v2 openstack-store.rgw.control
- Pool openstack-store.rgw.log requires crush rule update, device class is not set
  1) create a new replicated rule for pool:
    ceph osd crush create-replicated openstack-store.rgw.log_v2 default host hdd
  2) set pool crush rule to new one:
    ceph osd pool set crush_rule openstack-store.rgw.log openstack-store.rgw.log_v2
  3) check that replication is finished and status healthy: ceph -s
  4) in case of any problems revert step 2 and stop procedure:
    ceph osd pool set crush_rule openstack-store.rgw.log_v2 openstack-store.rgw.log
  5) remove standard (old) pool crush rule:
    ceph osd crush rule rm openstack-store.rgw.log
  6) rename new pool crush rule to standard name:
    ceph osd crush rule rename openstack-store.rgw.log_v2 openstack-store.rgw.log
- Pool openstack-store.rgw.buckets.non-ec requires crush rule update, device class is not set
  1) create a new replicated rule for pool:
    ceph osd crush create-replicated openstack-store.rgw.buckets.non-ec_v2 default host hdd
  2) set pool crush rule to new one:
    ceph osd pool set crush_rule openstack-store.rgw.buckets.non-ec openstack-store.rgw.buckets.non-ec_v2
  3) check that replication is finished and status healthy: ceph -s
  4) in case of any problems revert step 2 and stop procedure:
    ceph osd pool set crush_rule openstack-store.rgw.buckets.non-ec_v2 openstack-store.rgw.buckets.non-ec
  5) remove standard (old) pool crush rule:
    ceph osd crush rule rm openstack-store.rgw.buckets.non-ec
  6) rename new pool crush rule to standard name:
    ceph osd crush rule rename openstack-store.rgw.buckets.non-ec_v2 openstack-store.rgw.buckets.non-ec
- Pool .rgw.root requires crush rule update, device class is not set
  1) create a new replicated rule for pool:
    ceph osd crush create-replicated .rgw.root_v2 default host hdd
  2) set pool crush rule to new one:
    ceph osd pool set crush_rule .rgw.root .rgw.root_v2
  3) check that replication is finished and status healthy: ceph -s
  4) in case of any problems revert step 2 and stop procedure:
    ceph osd pool set crush_rule .rgw.root_v2 .rgw.root
  5) remove standard (old) pool crush rule:
    ceph osd crush rule rm .rgw.root
  6) rename new pool crush rule to standard name:
    ceph osd crush rule rename .rgw.root_v2 .rgw.root
- Pool openstack-store.rgw.meta requires crush rule update, device class is not set
  1) create a new replicated rule for pool:
    ceph osd crush create-replicated openstack-store.rgw.meta_v2 default host hdd
  2) set pool crush rule to new one:
    ceph osd pool set crush_rule openstack-store.rgw.meta openstack-store.rgw.meta_v2
  3) check that replication is finished and status healthy: ceph -s
  4) in case of any problems revert step 2 and stop procedure:
    ceph osd pool set crush_rule openstack-store.rgw.meta_v2 openstack-store.rgw.meta
  5) remove standard (old) pool crush rule:
    ceph osd crush rule rm openstack-store.rgw.meta
  6) rename new pool crush rule to standard name:
    ceph osd crush rule rename openstack-store.rgw.meta_v2 openstack-store.rgw.meta
- Pool openstack-store.rgw.buckets.index requires crush rule update, device class is not set
  1) create a new replicated rule for pool:
    ceph osd crush create-replicated openstack-store.rgw.buckets.index_v2 default host hdd
  2) set pool crush rule to new one:
    ceph osd pool set crush_rule openstack-store.rgw.buckets.index openstack-store.rgw.buckets.index_v2
  3) check that replication is finished and status healthy: ceph -s
  4) in case of any problems revert step 2 and stop procedure:
    ceph osd pool set crush_rule openstack-store.rgw.buckets.index_v2 openstack-store.rgw.buckets.index
  5) remove standard (old) pool crush rule:
    ceph osd crush rule rm openstack-store.rgw.buckets.index
  6) rename new pool crush rule to standard name:
    ceph osd crush rule rename openstack-store.rgw.buckets.index_v2 openstack-store.rgw.buckets.index
- Pool openstack-store.rgw.buckets.data requires crush rule update, device class is not set
  1) create a new erasure-coded profile:
    ceph osd erasure-code-profile set openstack-store_ecprofile_hdd crush-device-class=hdd crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=2 m=1 plugin=jerasure technique=reed_sol_van w=8
  2) create a new erasure-coded rule for pool:
    ceph osd crush create-erasure openstack-store.rgw.buckets.data_v2 openstack-store_ecprofile_hdd
  3) set pool crush rule to new one:
    ceph osd pool set crush_rule openstack-store.rgw.buckets.data openstack-store.rgw.buckets.data_v2
  4) check that replication is finished and status healthy: ceph -s
  5) in case of any problems revert step 2 and stop procedure:
    ceph osd pool set crush_rule openstack-store.rgw.buckets.data_v2 openstack-store.rgw.buckets.data
  6) remove standard (old) pool crush rule:
    ceph osd crush rule rm openstack-store.rgw.buckets.data
  7) rename new pool crush rule to standard name:
    ceph osd crush rule rename openstack-store.rgw.buckets.data_v2 openstack-store.rgw.buckets.data
  8) remove standard (old) erasure-coded profile:
    ceph osd erasure-code-profile rm openstack-store_ecprofile

Verify that the Ceph cluster has rebalanced and has the HEALTH_OK status:
```
ceph -s
```
Exit the ceph-tools Pod.

For CephFS, execute the rule-helper script to output the step-by-step instruction and run each step provided in the output manually.

Execution of the rule-helper script steps for CephFS

Get a shell of the ceph-tools Pod:

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash

Run the /tmp/rule-helper.py script with the following parameters:
```
python3 /tmp/rule-helper.py --prefix <cephfsName> --type cephfs --device-class <deviceClass>
```
Substitute the following parameters:
- <cephfsName> with CephFS name from spec.cephClusterSpec.sharedFilesystem.cephFS[0].name in the KaaSCephCluster object. In the example above, the name is cephfs-store.
- <deviceClass> with the device class selected in the previous steps.

Using the output of the command from the previous step, run manual commands step-by-step.

Example output for the hdd device class:

Found 3 rules require device class set
- Pool cephfs-store-metadata requires crush rule update, device class is not set
  1) create a new replicated rule for pool:
        ceph osd crush create-replicated cephfs-store-metadata_v2 default host ssd
  2) set pool crush rule to new one:
        ceph osd pool set crush_rule cephfs-store-metadata cephfs-store-metadata_v2
  3) check that replication is finished and status healthy: ceph -s
  4) in case of any problems revert step 2 and stop procedure:
        ceph osd pool set crush_rule cephfs-store-metadata_v2 cephfs-store-metadata
  5) remove standard (old) pool crush rule:
        ceph osd crush rule rm cephfs-store-metadata
  6) rename new pool crush rule to standard name:
        ceph osd crush rule rename cephfs-store-metadata_v2 cephfs-store-metadata
- Pool cephfs-store-default-pool requires crush rule update, device class is not set
  1) create a new replicated rule for pool:
        ceph osd crush create-replicated cephfs-store-default-pool_v2 default host ssd
  2) set pool crush rule to new one:
        ceph osd pool set crush_rule cephfs-store-default-pool cephfs-store-default-pool_v2
  3) check that replication is finished and status healthy: ceph -s
  4) in case of any problems revert step 2 and stop procedure:
        ceph osd pool set crush_rule cephfs-store-default-pool_v2 cephfs-store-default-pool
  5) remove standard (old) pool crush rule:
        ceph osd crush rule rm cephfs-store-default-pool
  6) rename new pool crush rule to standard name:
        ceph osd crush rule rename cephfs-store-default-pool_v2 cephfs-store-default-pool
- Pool cephfs-store-second-pool requires crush rule update, device class is not set
  1) create a new erasure-coded profile:
        ceph osd erasure-code-profile set cephfs-store-second-pool_ecprofile_ssd crush-device-class=ssd crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=2 m=1 plugin=jerasure technique=reed_sol_van w=8
  2) create a new erasure-coded rule for pool:
        ceph osd crush create-erasure cephfs-store-second-pool_v2 cephfs-store-second-pool_ecprofile_ssd
  3) set pool crush rule to new one:
        ceph osd pool set crush_rule cephfs-store-second-pool cephfs-store-second-pool_v2
  4) check that replication is finished and status healthy: ceph -s
  5) in case of any problems revert step 2 and stop procedure:
        ceph osd pool set crush_rule cephfs-store-second-pool_v2 cephfs-store-second-pool
  6) remove standard (old) pool crush rule:
        ceph osd crush rule rm cephfs-store-second-pool
  7) rename new pool crush rule to standard name:
        ceph osd crush rule rename cephfs-store-second-pool_v2 cephfs-store-second-pool
  8) remove standard (old) erasure-coded profile:
        ceph osd erasure-code-profile rm cephfs-store-second-pool_ecprofile

Verify that the Ceph cluster has rebalanced and has the HEALTH_OK status:
```
ceph -s
```
Exit the ceph-tools Pod.

Verify the pg_autoscaler module after switching deviceClass for all required pools:
```
ceph osd pool autoscale-status
```
The system response must contain all Ceph RGW and CephFS pools.

On the management cluster, edit the KaaSCephCluster object of the corresponding managed cluster by adding the selected device class to the deviceClass parameter of the updated Ceph RGW and CephFS pools:

kubectl -n <managedClusterProjectName> edit kaascephcluster

You can use this configuration step for further management of Ceph RGW and/or CephFS. It does not impact the existing Ceph cluster configuration.

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

StackLight¶

[31485] Elasticsearch Curator does not delete indices as per retention period¶

Fixed in 14.0.0(1) and 15.0.1

Note

If you obtain patch releases, the issue is addressed in 2.23.2 for management and regional clusters and in 11.7.1 and 12.7.1 for managed clusters.

Elasticsearch Curator does not delete any indices according to the configured retention period on any type of Container Cloud clusters.

To verify whether your cluster is affected:

Identify versions of Cluster releases installed on your clusters:

kubectl get cluster --all-namespaces \
-o custom-columns=CLUSTER:.metadata.name,NAMESPACE:.metadata.namespace,VERSION:.spec.providerSpec.value.release

The following list contains all affected Cluster releases:

mke-11-7-0-3-5-7
mke-13-4-4
mke-13-5-3
mke-13-6-0
mke-13-7-0
mosk-12-7-0-23-1

As a workaround, on the affected clusters, create a temporary CronJob for elasticsearch-curator to clean the required indices:

kubectl get cronjob elasticsearch-curator -n stacklight -o json \
| sed 's/5.7.6-[0-9]*/5.7.6-20230404082402/g' \
| jq '.spec.schedule = "30 * * * *"' \
| jq '.metadata.name = "temporary-elasticsearch-curator"' \
| jq 'del(.metadata.resourceVersion,.metadata.uid,.metadata.selfLink,.metadata.creationTimestamp,.metadata.annotations,.metadata.generation,.metadata.ownerReferences,.metadata.labels,.spec.jobTemplate.metadata.labels,.spec.jobTemplate.spec.template.metadata.creationTimestamp,.spec.jobTemplate.spec.template.metadata.labels)' \
| jq '.metadata.labels.app = "temporary-elasticsearch-curator"' \
| jq '.spec.jobTemplate.metadata.labels.app = "temporary-elasticsearch-curator"' \
| jq '.spec.jobTemplate.spec.template.metadata.labels.app = "temporary-elasticsearch-curator"' \
| kubectl create -f -

Note

This CronJob is removed automatically during upgrade to the major Container Cloud release 2.24.0 or to the patch Container Cloud release 2.23.3 if you obtain patch releases.

Components versions¶

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.23.0. For major components and versions of the Cluster release introduced in 2.23.0, see Cluster release 11.7.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

**Container Cloud release components versions**¶
Component	Application/Service	Version
Bare metal	ambassador ^Updated	1.23.3-alpine
	baremetal-operator ^Updated	base-focal-20230126095055
	baremetal-public-api ^Updated	1.36.3
	baremetal-provider ^Updated	1.36.5
	baremetal-resource-controller ^Updated	base-focal-20230130170757
	ironic ^Updated	yoga-focal-20230130125656
	kaas-ipam ^Updated	base-focal-20230127092754
	keepalived	0.19.0-5-g6a7e17d
	local-volume-provisioner ^Updated	2.5.0-1
	mariadb	10.6.7-focal-20221028120155
	metallb-controller	0.13.7
IAM	iam ^Updated	2.4.38
	iam-controller ^Updated	1.36.3
	keycloak	18.0.0
Container Cloud ^Updated	admission-controller	1.36.3
	agent-controller	1.36.3
	byo-credentials-controller	1.36.3
	byo-provider	1.36.3
	ceph-kcc-controller	1.36.3
	cert-manager	1.36.3
	client-certificate-controller	1.36.3
	event-controller	1.36.3
	golang	1.18.10
	kaas-public-api	1.36.3
	kaas-exporter	1.36.3
	kaas-ui	1.36.3
	license-controller	1.36.3
	lcm-controller	1.36.3
	machinepool-controller	1.36.3
	metrics-server	0.5.2
	mcc-cache	1.36.3
	portforward-controller	1.36.3
	proxy-controller	1.36.3
	rbac-controller	1.36.3
	release-controller	1.36.3
	rhellicense-controller	1.36.3
	scope-controller	1.36.3
	user-controller	1.36.3
OpenStack ^Updated	openstack-provider	1.36.3
	os-credentials-controller	1.36.3
VMware vSphere	metallb-controller	0.13.7
	vsphere-provider ^Updated	1.36.3
	vsphere-credentials-controller ^Updated	1.36.3
	keepalived	0.19.0-5-g6a7e17d
	squid-proxy ^Updated	0.0.1-8

Artifacts¶

This section lists the components artifacts of the Mirantis Container Cloud release 2.23.0. For artifacts of the Cluster release introduced in 2.23.0, see Cluster release 11.7.0.

Bare metal artifacts
Core artifacts
IAM artifacts

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶
Artifact	Component	Path
Binaries	baremetal-api ^Updated	https://binary.mirantis.com/core/helm/baremetal-api-1.36.3.tgz
	baremetal-operator ^Updated	https://binary.mirantis.com/core/helm/baremetal-operator-1.36.3.tgz
	baremetal-public-api ^Updated	https://binary.mirantis.com/core/helm/baremetal-public-api-1.36.3.tgz
	ironic-python-agent.initramfs ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20230126190304
	ironic-python-agent.kernel ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20230126190304
	kaas-ipam ^Updated	https://binary.mirantis.com/core/helm/kaas-ipam-1.36.3.tgz
	local-volume-provisioner ^Updated	https://binary.mirantis.com/core/helm/local-volume-provisioner-1.36.3.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.36.3.tgz
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-104-6e2e82c.tgz
Docker images	ambassador ^Updated	mirantis.azurecr.io/general/external/docker.io/library/nginx:1.23.3-alpine
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-focal-20230126095055
	baremetal-resource-controller ^Updated	mirantis.azurecr.io/bm/baremetal-resource-controller:base-focal-20230130170757
	dynamic_ipxe ^Updated	mirantis.azurecr.io/bm/dynamic-ipxe:base-focal-20230126202529
	dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-alpine-20230118150429
	dnsmasq-controller ^Updated	mirantis.azurecr.io/bm/dnsmasq-controller:base-focal-20230213185438
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:yoga-focal-20230130125656
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:yoga-focal-20230130125656
	ironic-prometheus-exporter ^Updated	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20221227163037
	kaas-ipam ^Updated	mirantis.azurecr.io/bm/kaas-ipam:base-focal-20230127092754
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.7-focal-20221028120155
	metallb-controller	mirantis.azurecr.io/bm/external/metallb/controller:v0.13.7-20221130155702
	metallb-speaker	mirantis.azurecr.io/bm/external/metallb/speaker:v0.13.7-20221130155702
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.19.0-5-g6a7e17d
	syslog-ng ^Updated	mirantis.azurecr.io/bm/syslog-ng:base-focal-20230126094812

Core artifacts¶
Artifact	Component	Paths
Bootstrap tarball ^Updated	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.36.4.tar.gz
	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.36.4.tar.gz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.36.3.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.36.3.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.36.5.tgz
	byo-credentials-controller	https://binary.mirantis.com/core/helm/byo-credentials-controller-1.36.3.tgz
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.36.3.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.36.3.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.36.3.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.36.3.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.36.3.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.36.3.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.36.3.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.36.3.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.36.3.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.36.3.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.36.3.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.36.3.tgz
	metrics-server	https://binary.mirantis.com/core/helm/metrics-server-1.36.3.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.36.3.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.36.3.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.36.3.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.36.3.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.36.3.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.36.3.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.36.3.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.36.3.tgz
	scope-controller	http://binary.mirantis.com/core/helm/scope-controller-1.36.3.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.36.3.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.36.3.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.36.3.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.36.3.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.36.3
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.36.3
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.36.3
	byo-credentials-controller ^Updated	mirantis.azurecr.io/core/byo-credentials-controller:1.36.3
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.36.3
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.6.1
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.36.3
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.36.3
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.36.3
	haproxy	mirantis.azurecr.io/lcm/mcc-haproxy:v0.17.0-8-g6ca89d5
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.36.3
	kaas-exporter	mirantis.azurecr.io/core/kaas-exporter:1.36.3
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.36.3
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:1.36.3
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.36.3
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.36.3
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.19.0-5-g6a7e17d
	metrics-server	mirantis.azurecr.io/core/external/metrics-server:v0.5.2
	nginx	mirantis.azurecr.io/core/external/nginx:1.36.3
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.36.3
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.36.3
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.36.3
	proxy-controller ^Updated	mirantis.azurecr.io/core/proxy-controller:1.36.3
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.36.3
	registry	mirantis.azurecr.io/lcm/registry:2.8.1
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.36.3
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.36.3
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.36.3
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-8
	storage-discovery ^Deprecated	mirantis.azurecr.io/core/storage-discovery:1.36.3
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.36.3
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.36.3
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.36.3

IAM artifacts¶
Artifact	Component	Path
Binaries	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
Helm charts	iam ^Updated	http://binary.mirantis.com/iam/helm/iam-2.4.38.tgz
	iam-proxy ^Updated	http://binary.mirantis.com/iam/helm/iam-proxy-0.2.14.tgz
	keycloak_proxy ^Removed	n/a
Docker images	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.0-20200311160233
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.7-focal-20220811085105
	keycloak ^Updated	mirantis.azurecr.io/iam/keycloak:0.5.14
	keycloak-gatekeeper	mirantis.azurecr.io/iam/keycloak-gatekeeper:7.1.3-3

Security notes¶

The table below contains the number of vendor-specific addressed CVEs with Critical or High severity.

In total, in the current Container Cloud release, 212 CVEs have been fixed and 16 artifacts (images) updated.

Addressed CVEs¶
Fixed CVE ID	# of updated artifacts
RHSA-2022:6206	20
RHSA-2022:4991	11
RHSA-2023:0838	8
RHSA-2022:7089	8
RHSA-2022:1065	8
RHSA-2022:0332	8
RHSA-2021:5082	8
RHSA-2021:2717	8
RHSA-2022:8638	7
RHSA-2022:6878	7
RHSA-2022:1642	7
RHSA-2022:0951	7
RHSA-2022:0658	7
RHSA-2022:1537	6
RHSA-2021:4903	5
RHSA-2020:3014	5
RHSA-2019:4114	5
RHSA-2022:6778	4
RHSA-2020:0575	4
RHSA-2022:5095	3
RHSA-2021:2359	3
RHSA-2023:0284	2
RHSA-2022:5056	2
RHSA-2022:4799	2
RHSA-2021:1206	2
RHSA-2019:0997	2
RHSA-2022:7192	1
RHSA-2021:2170	1
RHSA-2021:1989	1
RHSA-2021:1024	1
RHSA-2021:0670	1
RHSA-2020:5476	1
RHSA-2020:3658	1
RHSA-2020:2755	1
RHSA-2020:2637	1
RHSA-2020:2338	1
RHSA-2020:0902	1
RHSA-2020:0273	1
RHSA-2020:0271	1
RHSA-2019:2692	1
RHSA-2019:1714	1
RHSA-2019:1619	1
RHSA-2019:1145	1
CVE-2021-33574	18
CVE-2022-2068	7
CVE-2022-1664	7
CVE-2022-1292	7
CVE-2022-29155	6
CVE-2019-25013	6
CVE-2022-0778	5
CVE-2022-23219	4
CVE-2022-23218	4
CVE-2019-20916	4
CVE-2022-24407	3
CVE-2022-32207	2
CVE-2022-27404	2
CVE-2022-40023	1
CVE-2022-1941	1
CVE-2021-32839	1
CVE-2021-3711	1
CVE-2021-3517	1
ALAS2-2023-1915	1
ALAS2-2023-1911	1
ALAS2-2023-1908	1
ALAS2-2022-1902	2
ALAS2-2022-1885	1

The full list of the CVEs present in the current Container Cloud release is available at the Mirantis Security Portal.

See also

2.22.0¶

The Mirantis Container Cloud GA release 2.22.0:

Introduces support for the Cluster release 11.6.0 that is based on Mirantis Container Runtime 20.10.13 and Mirantis Kubernetes Engine 3.5.5 with Kubernetes 1.21.
Supports the Cluster release 12.5.0 that is based on the Cluster release 11.5.0 and represents Mirantis OpenStack for Kubernetes (MOSK) 22.5.
Does not support greenfield deployments on deprecated Cluster releases 11.5.0 and 8.10.0. Use the latest available Cluster releases of the series instead.

Caution

Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.22.0.

Enhancements¶

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.22.0. For the list of enhancements in the Cluster release 11.6.0 that is introduced by the Container Cloud release 2.22.0, see the Cluster releases (managed).

The ‘rebootRequired’ notification in the baremetal-based machine status
Custom network configuration for managed clusters based on Equinix Metal with private networking
Custom TLS certificates for the StackLight ‘iam-proxy’ endpoints
Cluster deployment and update history objects
Extended logging format for essential management cluster components

The ‘rebootRequired’ notification in the baremetal-based machine status¶

Added the rebootRequired field to the status of a Machine object for the bare metal provider. This field indicates whether a manual host reboot is required to complete the Ubuntu operating system updates, if any.

You can view this notification either using the Container Cloud API or web UI:

API: reboot.required.true in status:providerStatus of a Machine object
Web UI: the One or more machines require a reboot notification on the Clusters and Machines pages

Note

For MOSK-based deployments, the feature support is available since MOSK 23.1.

Learn more

Custom network configuration for managed clusters based on Equinix Metal with private networking¶

TechPreview

Implemented the ability to configure advanced network settings on managed clusters that are based on Equinix Metal with private networking. Using the custom parameter in the Cluster object, you can customize network configuration for the cluster machines. The feature comprises usage of dedicated Subnet and L2Template objects that contain necessary configuration for cluster machines.

Custom TLS certificates for the StackLight ‘iam-proxy’ endpoints¶

Implemented the ability to set up custom TLS certificates for the following StackLight iam-proxy endpoints on any type of Container Cloud clusters:

iam-proxy-alerta
iam-proxy-alertmanager
iam-proxy-grafana
iam-proxy-kibana
iam-proxy-prometheus

Learn more

Operations Guide: Configure TLS certificates for cluster applications

Cluster deployment and update history objects¶

Implemented the following Container Cloud objects describing the history of a cluster and machine deployment and update:

ClusterDeploymentStatus
ClusterUpgradeStatus
MachineDeploymentStatus
MachineUpgradeStatus

Using these objects, you can inspect cluster and machine deployment and update stages, their time stamps, statuses, and failure messages, if any. In the Container Cloud web UI, use the History option located under the More action icon of a cluster and machine.

For existing clusters, these objects become available after the management cluster upgrade to Container Cloud 2.22.0.

Learn more

Operations Guide: Inspect the history of a cluster deployment or update

Extended logging format for essential management cluster components¶

Extended the logging format for the admission-controller, storage-discovery, and all supported <providerName>-provider services of a management cluster. Now, log records for these services contain the following entries:

level:<debug,info,warn,error,panic>,
ts:<YYYY-MM-DDTHH:mm:ssZ>,
logger:<providerType>.<objectName>.req:<requestID>,
caller:<lineOfCode>,
msg:<message>,
error:<errorMessage>,
stacktrace:<codeInfo>

Addressed issues¶

The following issues have been addressed in the Mirantis Container Cloud release 2.22.0 along with the Cluster release 11.6.0:

[27192] Fixed the issue that prevented portforward-controller from accepting new connections correctly.
[26659] Fixed the issue that caused the deployment of a regional cluster based on bare metal or Equinix Metal with private networking to fail with mcc-cache Pods being stuck in the CrashLoopBackOff status of restarts.
[28783] Fixed the issue with Ceph condition getting stuck in absence of the Ceph cluster secrets information on the MOSK 22.3 clusters.

Caution

Starting from MOSK 22.4, the Ceph cluster version updates to 15.2.17. Therefore, if you applied the workaround for MOSK 22.3 described in Ceph known issue 28783, remove the version parameter definition from KaaSCephCluster after the managed cluster update to MOSK 22.4.
[26820] Fixed the issue with the status section in the KaaSCephCluster.status CR not reflecting issues during a Ceph cluster deletion.
[25624] Fixed the issue with inability to specify the Ceph pool API parameters by adding the parameters option that specifies the key-value map for the parameters of the Ceph pool.

Caution

For MKE clusters that are part of MOSK infrastructure, the feature support will become available in one of the following Container Cloud releases.
[28526] Fixed the issue with a low CPU limit 100m for kaas-exporter blocking metric collection.
[28134] Fixed the issue with failure to update a cluster with nodes being stuck in the Prepare state due to error when evicting pods for Patroni.
[27732-1] Fixed the issue with the OpenSearch elasticsearch.persistentVolumeClaimSize custom setting being overwritten by logging.persistentVolumeClaimSize during deployment of a Container Cloud cluster of any type and be set to the default 30Gi.

Depending on available resources on existing clusters that were affected by the issue, additional actions may be required after an update to Container Cloud 2.22.0. For details, see OpenSearchPVCMismatch alert raises due to the OpenSearch PVC size mismatch. New clusters deployed on top of Container Cloud 2.22.0 are not affected.
[27732-2] Fixed the issue with custom settings for the deprecated elasticsearch.logstashRetentionTime parameter being overwritten by the default setting set to 1 day.
[20876] Fixed the issue with StackLight Pods getting stuck with the Pod predicate NodeAffinity failed error due to the StackLight node label added to one machine and then removed from another one.
[28651] Updated Telemeter for StackLight to fix the discovered vulnerabilities.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.22.0 including the Cluster release 11.6.0.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal
Equinix Metal with private networking
vSphere
LCM
TLS configuration
StackLight
Ceph

Bare metal¶

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

[20736] Region deletion failure after regional deployment failure¶

If a baremetal-based regional cluster deployment fails before pivoting is done, the corresponding region deletion fails.

Workaround:

cluster
machine
baremetalhost
baremetalhostprofile
l2template
subnet
ipamhost
ipaddr

kubectl delete <objectName> -l kaas.mirantis.com/region=<regionName>

Warning

Do not use the same region name again after the regional cluster deployment failure since some objects that reference the region name may still exist.

Equinix Metal with private networking¶

[29296] Deployment of a managed cluster fails during provisioning¶

Deployment of a managed cluster based on Equinix Metal with private networking fails during provisioning with the following error:

InspectionError: Failed to obtain hardware details.
Ensure DHCP relay is up and running

Workaround:

In deployment/dnsmasq, udate the image tag version for the dhcpd container to base-alpine-20230118150429:
```
kubectl -n kaas edit deployment/dnsmasq
```

In dnsmasq.conf, override the default undionly.kpxe with the ipxe.pxe one:

kubectl -n kaas edit cm dnsmasq-config

Example of existing configuration:

dhcp-boot=/undionly.kpxe,httpd-http.ipxe.boot.local,dhcp-lb.ipxe.boot.local

Example of new configuration:

dhcp-boot=/ipxe.pxe,httpd-http.ipxe.boot.local,dhcp-lb.ipxe.boot.local

vSphere¶

[29647] The ‘Network prepared’ stage of cluster deployment never succeeds¶

Fixed in 11.7.0

During deployment of a vSphere-based management or regional cluster with IPAM disabled, the Network prepared stage gets stuck in the NotStarted status. The issue does not affect cluster deployment. Therefore, disregard the error message.

LCM¶

[5782] Manager machine fails to be deployed during node replacement¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a manager machine, the following problems may occur:

The system adds the node to Docker swarm but not to Kubernetes
The node Deployment gets stuck with failed RethinkDB health checks

Workaround:

Delete the failed node.
Wait for the MKE cluster to become healthy. To monitor the cluster status:
1. Log in to the MKE web UI as described in Connect to the Mirantis Kubernetes Engine web UI.
2. Monitor the cluster status as described in MKE Operations Guide: Monitor an MKE cluster with the MKE web UI.
Deploy a new node.

[5568] The calico-kube-controllers Pod fails to clean up resources¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During the unsafe or forced deletion of a manager machine running the calico-kube-controllers Pod in the kube-system namespace, the following issues occur:

The calico-kube-controllers Pod fails to clean up resources associated with the deleted node
The calico-node Pod may fail to start up on a newly created node if the machine is provisioned with the same IP address as the deleted machine had

As a workaround, before deletion of the node running the calico-kube-controllers Pod, cordon and drain the node:

kubectl cordon <nodeName>
kubectl drain <nodeName>

[27797] A cluster ‘kubeconfig’ stops working during MKE minor version update¶

As a workaround, during the cluster update process, use the admin kubeconfig instead of the existing one. Once the update completes, you can use the existing cluster kubeconfig again.

To obtain the admin kubeconfig:

kubectl --kubeconfig <pathToMgmtKubeconfig> get secret -n <affectedClusterNamespace> \
-o yaml <affectedClusterName>-kubeconfig | awk '/admin.conf/ {print $2}' | \
head -1 | base64 -d > clusterKubeconfig.yaml

If the related cluster is regional, replace <pathToMgmtKubeconfig> with <pathToRegionalKubeconfig>.

TLS configuration¶

[29604] The ‘failed to get kubeconfig’ error during TLS configuration¶

Fixed in 14.0.0(1) and 15.0.1

To verify the status of the TLS configuration being applied:

kubectl get cluster <ClusterName> -n <ClusterProjectName> -o jsonpath-as-json="{.status.providerStatus.tls.<Application>}"

Possible values for the <Application> parameter are as follows:

keycloak
ui
cache
mke
iamProxyAlerta
iamProxyAlertManager
iamProxyGrafana
iamProxyKibana
iamProxyPrometheus

Example of system response:

[
    {
        "expirationTime": "2024-01-06T09:37:04Z",
        "hostname": "domain.com",
    }
]

In this example, expirationTime equals the NotAfter field of the server certificate. And the value of hostname contains the configured application name.

StackLight¶

[30040] OpenSearch is not in the ‘deployed’ status during cluster update¶

Fixed in 11.7.0 and 12.7.0

Note

The issue may affect the Container Cloud or Cluster release update to the following versions:

2.22.0 for management and regional clusters
11.6.0 for management, regional, and managed clusters
13.2.5, 13.3.5, 13.4.3, and 13.5.2 for attached MKE clusters

The issue does not affect clusters originally deployed since the following Cluster releases: 11.0.0, 8.6.0, 7.6.0.

During cluster update to versions mentioned in the note above, the following OpenSearch-related error may occur on clusters that were originally deployed or attached using Container Cloud 2.15.0 or earlier, before the transition from Elasticsearch to OpenSearch:

The stacklight/opensearch release of the stacklight/stacklight-bundle HelmBundle
reconciled by the stacklight/stacklight-helm-controller Controller
is not in the "deployed" status for the last 15 minutes.

The issue affects clusters with elasticsearch.persistentVolumeClaimSize configured for values other than 30Gi.

To verify that the cluster is affected:

Verify whether the HelmBundleReleaseNotDeployed alert for the opensearch release is firing. If so, the cluster is most probably affected. Otherwise, the cluster is not affected.

Verify the reason of the HelmBundleReleaseNotDeployed alert for the opensearch release:

kubectl get helmbundle stacklight-bundle -n stacklight -o json | jq '.status.releaseStatuses[] | select(.chart == "opensearch") | .message'

Example system response from the affected cluster:

Upgrade "opensearch" failed: cannot patch "opensearch-master" with kind StatefulSet: \
StatefulSet.apps "opensearch-master" is invalid: spec: Forbidden: \
updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden

Workaround:

Scale down the opensearch-dashboards and metricbeat resources to 0:

kubectl -n stacklight scale --replicas 0 deployment opensearch-dashboards && \
kubectl -n stacklight get pods -l app=opensearch-dashboards | awk '{if (NR!=1) {print $1}}' | xargs -r \
kubectl -n stacklight wait --for=delete --timeout=10m pod

kubectl -n stacklight scale --replicas 0 deployment metricbeat && \
kubectl -n stacklight get pods -l app=metricbeat | awk '{if (NR!=1) {print $1}}' | xargs -r \
kubectl -n stacklight wait --for=delete --timeout=10m pod

Wait for the commands in this and next step to complete. The completion time depends on the cluster size.

Disable the elasticsearch-curator CronJob:

kubectl -n stacklight patch cronjobs elasticsearch-curator -p '{"spec": {"suspend": true}}'

Scale down the opensearch-master StatefulSet:

kubectl -n stacklight scale --replicas 0 statefulset opensearch-master && \
kubectl -n stacklight get pods -l app=opensearch-master | awk '{if (NR!=1) {print $1}}' | xargs -r \
kubectl -n stacklight wait --for=delete --timeout=30m pod

Delete the OpenSearch Helm release:

helm uninstall --no-hooks opensearch -n stacklight

Wait up to 5 minutes for Helm Controller to retry the upgrade and properly create the opensearch-master StatefulSet.

To verify readiness of the opensearch-master Pods:
```
kubectl -n stacklight wait --for=condition=Ready --timeout=30m pod -l app=opensearch-master
```
Example of a successful system response in an HA setup:
```
pod/opensearch-master-0 condition met
pod/opensearch-master-1 condition met
pod/opensearch-master-2 condition met
```
Example of a successful system response in an non-HA setup:
```
pod/opensearch-master-0 condition met
```

Scale up the opensearch-dashboards and metricbeat resources:

kubectl -n stacklight scale --replicas 1 deployment opensearch-dashboards && \
kubectl -n stacklight wait --for=condition=Ready --timeout=10m pod -l app=opensearch-dashboards

kubectl -n stacklight scale --replicas 1 deployment metricbeat && \
kubectl -n stacklight wait --for=condition=Ready --timeout=10m pod -l app=metricbeat

Enable the elasticsearch-curator CronJob:

kubectl -n stacklight patch cronjobs elasticsearch-curator -p '{"spec": {"suspend": false}}'

[29329] Recreation of the Patroni container replica is stuck¶

Fixed in 11.7.0 and 12.7.0

During an update of a Container Cloud cluster of any type, recreation of the Patroni container replica is stuck in the degraded state due to the liveness probe killing the container that runs the pg_rewind procedure. The issue affects clusters on which the pg_rewind procedure takes more time than the full cycle of the liveness probe.

The sample logs of the affected cluster:

INFO: doing crash recovery in a single user mode
ERROR: Crash recovery finished with code=-6
INFO:  stdout=
INFO:  stderr=2023-01-11 10:20:34 GMT [64]: [1-1] 63be8d72.40 0     LOG:  database system was interrupted; last known up at 2023-01-10 17:00:59 GMT
[64]: [2-1] 63be8d72.40 0  LOG:  could not read from log segment 00000002000000000000000F, offset 0: read 0 of 8192
[64]: [3-1] 63be8d72.40 0  LOG:  invalid primary checkpoint record
[64]: [4-1] 63be8d72.40 0  PANIC:  could not locate a valid checkpoint record

Workaround:

For the affected replica and PVC, run:

kubectl delete persistentvolumeclaim/storage-volume-patroni-<replica-id> -n stacklight

kubectl delete pod/patroni-<replica-id> -n stacklight

[28822] Reference Application triggers alerts during its upgrade¶

Fixed in 11.7.0 and 12.7.0

On managed clusters with enabled Reference Application, the following alerts are triggered during a managed cluster update from the Cluster release 11.5.0 to 11.6.0 or 7.11.0 to 11.5.0:

KubeDeploymentOutage for the refapp Deployment
RefAppDown
RefAppProbeTooLong
RefAppTargetDown

This behavior is expected, no actions are required. Therefore, disregard these alerts.

[28479] Increase of the ‘metric-collector’ Pod restarts due to OOM¶

Fixed in 11.7.0 and 12.7.0

On the baremetal-based management clusters, the restarts count of the metric-collector Pod is increased in time with reason: OOMKilled in the containerStatuses of the metric-collector Pod. Only clusters with HTTP proxy enabled are affected.

Such behavior is expected. Therefore, disregard these restarts.

[28373] Alerta can get stuck after a failed initialization¶

Fixed in 11.7.0 and 12.7.0

During creation of a Container Cloud cluster of any type with StackLight enabled, Alerta can get stuck after a failed initialization with only 1 Pod in the READY state. For example:

kubectl get po -n stacklight -l app=alerta

NAME                          READY   STATUS    RESTARTS   AGE
pod/alerta-5f96b775db-45qsz   1/1     Running   0          20h
pod/alerta-5f96b775db-xj4rl   0/1     Running   0          20h

Workaround:

Recreate the affected Alerta Pod:

kubectl --kubeconfig <affectedClusterKubeconfig> -n stacklight delete pod <stuckAlertaPodName>

Verify that both Alerta Pods are in the READY state:

kubectl get po -n stacklight -l app=alerta

[20876] StackLight pods get stuck with the ‘NodeAffinity failed’ error¶

Note

Moving forward, the workaround for this issue will be moved from Release Notes to Operations Guide: Troubleshoot StackLight.

On a managed cluster, the StackLight pods may get stuck with the Pod predicate NodeAffinity failed error in the pod status. The issue may occur if the StackLight node label was added to one machine and then removed from another one.

The issue does not affect the StackLight services, all required StackLight pods migrate successfully except extra pods that are created and stuck during pod migration.

As a workaround, remove the stuck pods:

kubectl --kubeconfig <managedClusterKubeconfig> -n stacklight delete pod <stuckPodName>

Ceph¶

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

Components versions¶

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.22.0. For major components and versions of the Cluster release introduced in 2.22.0, see Cluster release 11.6.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

**Container Cloud release components versions**¶
Component	Application/Service	Version
AWS ^Updated	aws-provider	1.35.11
	aws-credentials-controller	1.35.11
Azure ^Updated	azure-provider	1.35.11
	azure-credentials-controller	1.35.11
Bare metal	ambassador	1.20.1-alpine
	baremetal-operator ^Updated	base-focal-20221130142939
	baremetal-public-api ^Updated	1.35.11
	baremetal-provider ^Updated	1.35.11
	baremetal-resource-controller	base-focal-20221219124546
	ironic ^Updated	yoga-focal-20221118093824
	kaas-ipam	base-focal-20221202191902
	keepalived	0.19.0-5-g6a7e17d
	local-volume-provisioner ^Updated	2.5.0-1
	mariadb ^Updated	10.6.7-focal-20221028120155
	metallb-controller ^Updated	0.13.7
IAM	iam ^Updated	2.4.36
	iam-controller ^Updated	1.35.11
	keycloak	18.0.0
Container Cloud ^Updated	admission-controller	1.35.12
	agent-controller	1.35.11
	byo-credentials-controller	1.35.11
	byo-provider	1.35.11
	ceph-kcc-controller	1.35.11
	cert-manager	1.35.11
	client-certificate-controller	1.35.11
	event-controller	1.35.11
	golang ^Updated	1.18.8
	kaas-public-api	1.35.11
	kaas-exporter	1.35.11
	kaas-ui	1.35.11
	license-controller	1.35.11
	lcm-controller	0.3.0-352-gf55d6378
	machinepool-controller	1.35.11
	mcc-cache	1.35.11
	metrics-server	0.5.2
	portforward-controller	1.35.11
	proxy-controller	1.35.11
	rbac-controller	1.35.11
	release-controller	1.35.11
	rhellicense-controller	1.35.11
	scope-controller	1.35.11
	user-controller	1.35.11
Equinix Metal	equinix-provider ^Updated	1.35.11
	equinix-credentials-controller ^Updated	1.35.11
	keepalived	0.19.0-5-g6a7e17d
OpenStack ^Updated	openstack-provider	1.35.11
	os-credentials-controller	1.35.11
VMware vSphere	metallb-controller	0.13.7
	vsphere-provider ^Updated	1.35.11
	vsphere-credentials-controller ^Updated	1.35.11
	keepalived	0.19.0-5-g6a7e17d
	squid-proxy ^Updated	0.0.1-8

Artifacts¶

This section lists the components artifacts of the Mirantis Container Cloud release 2.22.0. For artifacts of the Cluster release introduced in 2.22.0, see Cluster release 11.6.0.

Bare metal artifacts
Core artifacts
IAM artifacts

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶
Artifact	Component	Path
Binaries	baremetal-api ^Updated	https://binary.mirantis.com/core/helm/baremetal-api-1.35.11.tgz
	baremetal-operator ^Updated	https://binary.mirantis.com/core/helm/baremetal-operator-1.35.11.tgz
	baremetal-public-api ^Updated	https://binary.mirantis.com/core/helm/baremetal-public-api-1.35.11.tgz
	ironic-python-agent.initramfs ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20221228205257
	ironic-python-agent.kernel ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20221228205257
	kaas-ipam ^Updated	https://binary.mirantis.com/core/helm/kaas-ipam-1.35.11.tgz
	local-volume-provisioner ^Updated	https://binary.mirantis.com/core/helm/local-volume-provisioner-1.35.11.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.35.11.tgz
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-104-6e2e82c.tgz
Docker images	ambassador	mirantis.azurecr.io/general/external/docker.io/library/nginx:1.20.1-alpine
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-focal-20221130142939
	baremetal-resource-controller ^Updated	mirantis.azurecr.io/bm/baremetal-resource-controller:base-focal-20221219124546
	dynamic_ipxe ^Updated	mirantis.azurecr.io/bm/dynamic-ipxe:base-focal-20221219135753
	dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-alpine-20221121215534
	dnsmasq-controller ^Updated	mirantis.azurecr.io/bm/dnsmasq-controller:base-focal-20221219112845
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:yoga-focal-20221118093824
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:yoga-focal-20221118093824
	ironic-prometheus-exporter ^Updated	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20221117115942
	kaas-ipam ^Updated	mirantis.azurecr.io/bm/kaas-ipam:base-focal-20221202191902
	mariadb ^Updated	mirantis.azurecr.io/general/mariadb:10.6.7-focal-20221028120155
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.19.0-5-g6a7e17d
	metallb-controller ^Updated	mirantis.azurecr.io/bm/external/metallb/controller:v0.13.7-20221130155702
	metallb-speaker ^Updated	mirantis.azurecr.io/bm/external/metallb/speaker:v0.13.7-20221130155702
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-focal-20220128103433

Core artifacts¶
Artifact	Component	Paths
Bootstrap tarball ^Updated	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.35.11.tar.gz
	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.35.11.tar.gz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.35.11.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.35.11.tgz
	aws-credentials-controller	https://binary.mirantis.com/core/helm/aws-credentials-controller-1.35.11.tgz
	aws-provider	https://binary.mirantis.com/core/helm/aws-provider-1.35.11.tgz
	azure-credentials-controller	https://binary.mirantis.com/core/helm/azure-credentials-controller-1.35.11.tgz
	azure-provider	https://binary.mirantis.com/core/helm/azure-provider-1.35.11.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.35.11.tgz
	byo-credentials-controller	https://binary.mirantis.com/core/helm/byo-credentials-controller-1.35.11.tgz
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.35.11.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.35.11.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.35.11.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.35.11.tgz
	configuration-collector	https://binary.mirantis.com/core/helm/configuration-collector-1.35.11.tgz
	equinix-credentials-controller	https://binary.mirantis.com/core/helm/equinix-credentials-controller-1.35.11.tgz
	equinix-provider	https://binary.mirantis.com/core/helm/equinix-provider-1.35.11.tgz
	equinixmetalv2-provider	https://binary.mirantis.com/core/helm/equinixmetalv2-provider-1.35.11.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.35.11.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.35.11.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.35.11.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.35.11.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.35.11.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.35.11.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.35.11.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.35.11.tgz
	metrics-server	https://binary.mirantis.com/core/helm/metrics-server-1.35.11.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.35.11.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.35.11.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.35.11.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.35.11.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.35.11.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.35.11.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.35.11.tgz
	scope-controller	http://binary.mirantis.com/core/helm/scope-controller-1.35.11.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.35.11.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.35.11.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.35.11.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.35.11.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.35.11
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.35.11
	aws-cluster-api-controller ^Updated	mirantis.azurecr.io/core/aws-cluster-api-controller:1.35.11
	aws-credentials-controller ^Updated	mirantis.azurecr.io/core/aws-credentials-controller:1.35.11
	azure-cloud-controller-manager ^New	mirantis.azurecr.io/lcm/external/azure-cloud-controller-manager:v1.23.11
	azure-cloud-node-manager ^New	mirantis.azurecr.io/lcm/external/azure-cloud-node-manager:v1.23.11
	azure-cluster-api-controller ^Updated	mirantis.azurecr.io/core/azure-cluster-api-controller:1.35.11
	azure-credentials-controller ^Updated	mirantis.azurecr.io/core/azure-credentials-controller:1.35.11
	azuredisk-csi ^New	mirantis.azurecr.io/lcm/azuredisk-csi-driver:v0.20.0-25-gfaef237
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.35.11
	byo-credentials-controller ^Updated	mirantis.azurecr.io/core/byo-credentials-controller:1.35.11
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.35.11
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.6.1
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.35.11
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.35.11
	cluster-api-provider-equinix ^Updated	mirantis.azurecr.io/core/equinix-cluster-api-controller:1.35.11
	equinix-credentials-controller ^Updated	mirantis.azurecr.io/core/equinix-credentials-controller:1.35.11
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.35.11
	haproxy	mirantis.azurecr.io/lcm/mcc-haproxy:v0.17.0-8-g6ca89d5
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.35.11
	kaas-exporter	mirantis.azurecr.io/core/kaas-exporter:1.35.11
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.35.11
	lcm-controller ^Updated	mirantis.azurecr.io/lcm/lcm-controller:v0.3.0-352-gf55d6378
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.35.11
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.35.11
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.19.0-5-g6a7e17d
	metrics-server	mirantis.azurecr.io/core/external/metrics-server:v0.5.2
	nginx	mirantis.azurecr.io/core/external/nginx:1.35.11
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.35.11
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.35.11
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.35.11
	proxy-controller ^Updated	mirantis.azurecr.io/core/proxy-controller:1.35.11
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.35.11
	registry ^Updated	mirantis.azurecr.io/lcm/registry:2.8.1
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.35.11
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.35.11
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.35.11
	squid-proxy ^Updated	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-8
	storage-discovery	mirantis.azurecr.io/core/storage-discovery:1.35.11
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.35.11
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.35.11
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.35.11

IAM artifacts¶
Artifact	Component	Path
Binaries	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
Helm charts	iam ^Updated	http://binary.mirantis.com/iam/helm/iam-2.4.36.tgz
	iam-proxy	http://binary.mirantis.com/iam/helm/iam-proxy-0.2.13.tgz
	keycloak_proxy ^Updated	http://binary.mirantis.com/core/helm/keycloak_proxy-1.35.11.tgz
Docker images	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.0-20200311160233
	mariadb	mirantis.azurecr.io/general/mariadb:10.6.7-focal-20220811085105
	keycloak ^Updated	mirantis.azurecr.io/iam/keycloak:0.5.13
	keycloak-gatekeeper	mirantis.azurecr.io/iam/keycloak-gatekeeper:7.1.3-3

Security notes¶

The table below contains the number of vendor-specific addressed CVEs with Critical or High severity.

In total, in the current Container Cloud release, 6 CVEs have been fixed and 4 artifacts (images) updated.

Addressed CVEs¶
Fixed CVE ID	# of updated artifacts
CVE-2022-40023	2
CVE-2022-25236	1
CVE-2022-25235	1
RHSA-2022:8638	1
RHSA-2022:7089	1
RHSA-2022:6878	1

The full list of the CVEs present in the current Container Cloud release is available at the Mirantis Security Portal.

See also

Releases delivered in 2022¶

This section contains historical information on the unsupported Container Cloud releases delivered in 2022. For the latest supported Container Cloud release, see Container Cloud releases.

Unsupported Container Cloud releases 2022¶
Version	Release date	Summary
2.21.1	Dec 19, 2022	Based on 2.21.0, Container Cloud 2.21.1: Introduces the Cluster release 12.5.0 that is based on 11.5.0 and supports Mirantis OpenStack for Kubernetes (MOSK) 22.5. Supports the Cluster releases 11.5.0 and 7.11.0. The deprecated Cluster releases 11.4.0, 8.10.0, and 7.10.0 are not supported for new deployments. Contains features and amendments of the parent release 2.21.0.
2.21.0	Nov 22, 2022	MKE patch releases update from 3.4.10 to 3.4.11 and from 3.5.4 to 3.5.5 MCR patch release update from 20.10.12 to 20.10.13 MetalLB minor version update from 0.12.1 to 0.13.4 `BareMetalHostCredential` CR Dnsmasq configuration enhancements Combining router and seed node settings on a single Equinix Metal server Graceful machine deletion Container Cloud web UI support for custom Docker registries Enhanced etcd monitoring Reference Application for workload monitoring Ceph secrets specification in the Ceph cluster status Amazon S3 bucket policies for Ceph Object Storage users Documentation: Firewall configuration
2.20.1	Sep 29, 2022	Based on 2.20.0, Container Cloud 2.20.1: Introduces the Cluster release 8.10.0 that is based on 7.10.0 and supports Mirantis OpenStack for Kubernetes (MOSK) 22.4. Supports the Cluster releases 7.10.0 and 11.4.0. The deprecated Cluster releases 11.3.0, 8.8.0, and 7.9.0 are not supported for new deployments. Contains features and amendments of the parent release 2.20.0.
2.20.0	Sep 5, 2022	MKE and MCR versions update Configuration of TLS certificates for `mcc-cache` and MKE General availability support for MITM proxy Bastion node configuration for OpenStack and AWS managed clusters New `member` role for IAM Bare metal: Mandatory IPAM service label for bare metal LCM subnets Flexible size units for bare metal host profiles Ceph: Ceph removal from management and regional clusters Creation of Ceph RADOS Gateway users Custom RBD map options Ceph Manager modules configuration Ceph daemons health check configuration
2.19.0	July 27, 2022	Modification of network configuration on existing machines New format of log entries on management clusters Extended and basic versions of logs Removal of Helm v2 support in Helm Controller StackLight: Kubernetes Containers Grafana dashboard Improvements to alerting Ceph: Ceph OSD removal or replacement by ID Multiple Ceph data pools per CephFS Container Cloud web UI: Upgrade order for machines Booting an OpenStack machine from a volume Distribution selector for bare metal machines Elasticsearch switch to OpenSearch Ceph cluster summary
2.18.1	June 30, 2022	Based on 2.18.0, Container Cloud 2.18.1: Introduces the Cluster release 8.8.0 that is based on 7.8.0 and supports Mirantis OpenStack for Kubernetes (MOSK) 22.3. Supports the Cluster releases 7.8.0 and 11.2.0. The deprecated Cluster releases 11.1.0, 8.6.0, and 7.7.0 are not supported for new deployments. Contains features and amendments of the parent release 2.18.0.
2.18.0	June 13, 2022	MKE and MCR version update Ubuntu kernel update for bare metal clusters Support for Ubuntu 20.04 on greenfield vSphere deployments Booting a machine from a block storage volume for OpenStack provider IPSec encryption for Kubernetes networking Support for MITM proxy Support for custom Docker registries Upgrade sequence for machines Deprecation of public network mode on the Equinix Metal based deployments Enablement of Salesforce propagation to all clusters using web UI StackLight: Elasticsearch switch to OpenSearch Improvements to StackLight alerting Prometheus remote write StackLight mandatory parameters Ceph daemons placement Documentation enhancements
2.17.0	May 11, 2022	General availability for Ubuntu 20.04 on greenfield deployments EBS instead of NVMe as persistent storage for AWS-based nodes Container Cloud on top of MOSK Victoria with Tungsten Fabric MKE 3.5.1 for management and regional clusters Manager nodes deletion on all cluster types Automatic propagation of Salesforce configuration to all clusters Custom values for node labels Machine pools StackLight: Elasticsearch retention time per index Helm controller monitoring Ceph: Configurable timeouts for Ceph requests Configurable replicas count for Ceph controllers KaaSCephCluster controller
2.16.1	Apr 14, 2022	Based on 2.16.0, Container Cloud 2.16.1: Introduces the Cluster release 8.6.0 that is based on 7.6.0 and supports Mirantis OpenStack for Kubernetes (MOSK) 22.2. Supports the Cluster releases 7.6.0 and 11.0.0. The deprecated Cluster releases 8.5.0, 7.5.0, and 5.22.0 are not supported for new deployments. Contains features and amendments of the parent release 2.16.0
2.16.0	Mar 31, 2022	Support for MKE 3.5.1 and MKE version update from 3.4.6 to 3.4.7 Automatic renewal of internal TLS certificates Keepalived for built-in load balancing in standalone containers Reworked ‘Reconfigure’ phase of LCMMachine Bare metal provider: Ubuntu 20.04 for greenfield bare metal managed cluster Additional regional cluster on bare metal MOSK on local RAID devices Any interface name for bare metal LCM network StackLight: Improvements to StackLight alerting Elasticsearch retention time per index Prometheus Blackbox Exporter configuration Custom Prometheus scrape configurations Elasticsearch switch to OpenSearch Container Cloud web UI: License management Scheduling of a management cluster upgrade
2.15.1	Feb 23, 2022	Based on 2.15.0, this release introduces the Cluster release 8.5.0 that is based on 5.22.0 and supports Mirantis OpenStack for Kubernetes (MOSK) 22.1. For the list of Cluster releases 7.x and 5.x that are supported by 2.15.1 as well as for its features with addressed and known issues, refer to the parent release 2.15.0.
2.15.0	Jan 31, 2022	MCR version update from 20.10.6 to 20.10.8 Scheduled Container Cloud auto-upgrade Cluster and machine maintenance mode Improvements for monitoring of machine deployment live status Deprecation of iam-api and IAM CLI HAProxy instead of NGINX for vSphere, Equinix Metal, and bare metal providers Additional regional cluster on Equinix Metal with private networking as Technology Preview Bare metal: Automatic upgrade of bare metal host operating system during cluster update Dedicated subnet for externally accessible Kubernetes API endpoint Ceph: Automated Ceph LCM Ceph CSI provisioner tolerations and node affinity KaaSCephCluster.status enhancement Shared File System (CephFS) Switch of Ceph Helm releases from v2 to v3 StackLight: Node Exporter collectors Improvements to StackLight alerting Metric Collector alerts Documentation: Expanding the capacity of the existing `Subnet` resources on a running cluster Calculating target ratio for Ceph pools

2.21.1¶

The Mirantis Container Cloud GA release 2.21.1 is based on 2.21.0 and:

Introduces support for the Cluster release 12.5.0 that is based on the Cluster release 11.5.0 and represents Mirantis OpenStack for Kubernetes (MOSK) 22.5.
Introduces support for Mirantis Kubernetes Engine 3.5.5 with Kubernetes 1.21 and Mirantis Container Runtime 20.10.13 in the 12.x Cluster release series.
Supports the latest Cluster releases 7.11.0 and 11.5.0.
Does not support greenfield deployments based on deprecated Cluster releases 11.4.0, 8.10.0, and 7.10.0. Use the latest available Cluster releases of the series instead.

For details about the Container Cloud release 2.21.1, refer to its parent release 2.21.0:

Caution

See also

2.21.0¶

The Mirantis Container Cloud GA release 2.21.0:

Introduces support for the Cluster release 11.5.0 that is based on Mirantis Container Runtime 20.10.13 and Mirantis Kubernetes Engine 3.5.5 with Kubernetes 1.21.
Introduces support for the Cluster release 7.11.0 that is based on Mirantis Container Runtime 20.10.13 and Mirantis Kubernetes Engine 3.4.11 with Kubernetes 1.20.
Supports the Cluster release 8.10.0 that is based on the Cluster release 7.10.0 and represents Mirantis OpenStack for Kubernetes (MOSK) 22.4.
Does not support greenfield deployments on deprecated Cluster releases 11.4.0, 8.8.0, and 7.10.0. Use the latest available Cluster releases of the series instead.

Caution

Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.21.0.

Caution

Container Cloud 2.21.0 requires manual post-upgrade steps. For details, see Post-upgrade actions.

Enhancements¶

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.21.0. For the list of enhancements in the Cluster releases 11.5.0 and 7.11.0 that are introduced by the Container Cloud release 2.21.0, see the Cluster releases (managed).

‘BareMetalHostCredential’ custom resource for bare metal hosts
Dnsmasq configuration enhancements
Combining router and seed node settings on one Equinix Metal server
Graceful machine deletion
Add custom Docker registries using the Container Cloud web UI

‘BareMetalHostCredential’ custom resource for bare metal hosts¶

Implemented the BareMetalHostCredential custom resource to simplify permissions and roles management on a bare metal management, regional, and managed cluster.

Note

For MOSK-based deployments, the feature support is available since MOSK 22.5.

The BareMetalHostCredential object creation triggers the following automatic actions:

Create an underlying Secret object containing data about username and password of the bmc account of the related BareMetalHostCredential object.
Erase sensitive password data of the bmc account from the BareMetalHostCredential object.
Add the created Secret object name to the spec.password.name section of the related BareMetalHostCredential object.
Update BareMetalHost.spec.bmc.credentialsName with the BareMetalHostCredential object name.

Note

When you delete a BareMetalHost object, the related BareMetalHostCredential object is deleted automatically.

Note

On existing clusters, a BareMetalHostCredential object is automatically created for each BareMetalHost object during a cluster update.

Learn more

Dnsmasq configuration enhancements¶

Enhanced the logic of the dnsmasq server to listen on the PXE network of the management cluster by using the dhcp-lb Kubernetes Service instead of listening on the PXE interface of one management cluster node.

To configure the DHCP relay service, specify the external address of the dhcp-lb Kubernetes Service as an upstream address for the relayed DHCP requests, which is the IP helper address for DHCP. There is the dnsmasq Deployment behind this service that can only accept relayed DHCP requests.

Container Cloud has its own DHCP relay running on one of the management cluster nodes. That DHCP relay serves for proxying DHCP requests in the same L2 domain where the management cluster nodes are located.

The enhancement comprises deprecation of the dnsmasq.dhcp_range parameter. Use the Subnet object configuration for this purpose instead.

Note

If you configured multiple DHCP ranges before Container Cloud 2.21.0 during the management cluster bootstrap, the DHCP configuration will automatically migrate to Subnet objects after cluster upgrade to 2.21.0.

Caution

Using of custom DNS server addresses for servers that boot over PXE is not supported.

Learn more

Combining router and seed node settings on one Equinix Metal server¶

Implemented the ability to combine configuration of a router and seed node on the same server when preparing infrastructure for an Equinix Metal based Container Cloud with private networking using Terraform templates. Set router_as_seed to true in the required Metro configuration while preparing terraform.tfvars to combine both the router and seed node roles.

Learn more

Container Cloud on Equinix Metal templates

Graceful machine deletion¶

TechPreview

Implemented the possibility to safely clean up a node resources using the Container Cloud API before deleting it from a cluster. Using the deletionPolicy: graceful parameter in the providerSpec.value section of the Machine object, the cloud provider controller now prepares a machine for deletion by cordoning, draining, and removing the related node from Docker Swarm. If required, you can abort a machine deletion when using deletionPolicy: graceful, but only before the related node is removed from Docker Swarm.

Caution

For MKE clusters that are part of MOSK infrastructure, the feature support will become available in one of the following Container Cloud releases.

Learn more

Delete a cluster machine using CLI

Add custom Docker registries using the Container Cloud web UI¶

Enhanced support for custom Docker registries configuration in management, regional, and managed clusters by adding the Container Registries tab to the Container Cloud web UI. Using this tab, you can configure CA certificates on machines to access private Docker registries.

Note

For MOSK-based deployments, the feature support is available since MOSK 22.5.

Learn more

Define a custom CA certificate for a private Docker registry

Addressed issues¶

The following issues have been addressed in the Mirantis Container Cloud release 2.21.0 along with the Cluster releases 11.5.0 and 7.11.0:

[23002] Fixed the issue with inability to set a custom value for a predefined node label using the Container Cloud web UI.
[26416] Fixed the issue with inability to automatically upload an MKE client bundle during cluster attachment using the Container Cloud web UI.
[26740] Fixed the issue with failure to upgrade a management cluster with a Keycloak or web UI TLS custom certificate.
[27193] Fixed the issue with missing permissions for the m:kaas:<namespaceName>@member role that are required for the Container Cloud web UI to work properly. The issue relates to reading permissions for resources objects of all providers as well as clusterRelease, unsupportedCluster objects, and so on.
[26379] Fixed the issue with missing logs for MOSK-related namespaces when using the container-cloud collect logs command without the --extended flag.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.21.0 including the Cluster releases 11.5.0 and 7.11.0.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

MKE
Bare metal
Equinix Metal with private networking
vSphere
LCM
StackLight
Storage

MKE¶

[20651] A cluster deployment or update fails with not ready compose deployments¶

A managed cluster deployment, attachment, or update to a Cluster release with MKE versions 3.3.13, 3.4.6, 3.5.1, or earlier may fail with the compose pods flapping (ready > terminating > pending) and with the following error message appearing in logs:

'not ready: deployments: kube-system/compose got 0/0 replicas, kube-system/compose-api
 got 0/0 replicas'
 ready: false
 type: Kubernetes

Workaround:

Disable Docker Content Trust (DCT):
1. Access the MKE web UI as admin.
2. Navigate to Admin > Admin Settings.
3. In the left navigation pane, click Docker Content Trust and disable it.
Restart the affected deployments such as calico-kube-controllers, compose, compose-api, coredns, and so on:
```
kubectl -n kube-system delete deployment <deploymentName>
```
Once done, the cluster deployment or update resumes.
Re-enable DCT.

Bare metal¶

[26659] Regional cluster deployment failure with stuck ‘mcc-cache’ Pods¶

Fixed in 11.6.0

Deployment of a regional cluster based on bare metal or Equinix Metal with private networking fails with mcc-cache Pods being stuck in the CrashLoopBackOff status of restarts.

As a workaround, remove failed mcc-cache Pods to restart them automatically. For example:

kubectl -n kaas delete pod mcc-cache-0

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

[20736] Region deletion failure after regional deployment failure¶

If a baremetal-based regional cluster deployment fails before pivoting is done, the corresponding region deletion fails.

Workaround:

cluster
machine
baremetalhost
baremetalhostprofile
l2template
subnet
ipamhost
ipaddr

kubectl delete <objectName> -l kaas.mirantis.com/region=<regionName>

Warning

Do not use the same region name again after the regional cluster deployment failure since some objects that reference the region name may still exist.

Equinix Metal with private networking¶

[26659] Regional cluster deployment failure with stuck ‘mcc-cache’ Pods¶

Fixed in 11.6.0

Deployment of a regional cluster based on bare metal or Equinix Metal with private networking fails with mcc-cache Pods being stuck in the CrashLoopBackOff status of restarts.

As a workaround, remove failed mcc-cache Pods to restart them automatically. For example:

kubectl -n kaas delete pod mcc-cache-0

vSphere¶

[26070] RHEL system cannot be registered in Red Hat portal over MITM proxy¶

Deployment of RHEL machines using the Red Hat portal registration, which requires user and password credentials, over MITM proxy fails while building the virtual machines template with the following error:

Unable to verify server's identity: [SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed (_ssl.c:618)

The Container Cloud deployment gets stuck while applying the RHEL license to machines with the same error in the lcm-agent logs.

As a workaround, use the internal Red Hat Satellite server that a VM can access directly without a MITM proxy.

LCM¶

[5782] Manager machine fails to be deployed during node replacement¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During replacement of a manager machine, the following problems may occur:

The system adds the node to Docker swarm but not to Kubernetes
The node Deployment gets stuck with failed RethinkDB health checks

Workaround:

Delete the failed node.
Wait for the MKE cluster to become healthy. To monitor the cluster status:
1. Log in to the MKE web UI as described in Connect to the Mirantis Kubernetes Engine web UI.
2. Monitor the cluster status as described in MKE Operations Guide: Monitor an MKE cluster with the MKE web UI.
Deploy a new node.

[5568] The calico-kube-controllers Pod fails to clean up resources¶

Fixed in 2.28.4 (17.3.4 and 16.3.4)

During the unsafe or forced deletion of a manager machine running the calico-kube-controllers Pod in the kube-system namespace, the following issues occur:

The calico-kube-controllers Pod fails to clean up resources associated with the deleted node
The calico-node Pod may fail to start up on a newly created node if the machine is provisioned with the same IP address as the deleted machine had

As a workaround, before deletion of the node running the calico-kube-controllers Pod, cordon and drain the node:

kubectl cordon <nodeName>
kubectl drain <nodeName>

[27797] A cluster ‘kubeconfig’ stops working during MKE minor version update¶

As a workaround, during the cluster update process, use the admin kubeconfig instead of the existing one. Once the update completes, you can use the existing cluster kubeconfig again.

To obtain the admin kubeconfig:

kubectl --kubeconfig <pathToMgmtKubeconfig> get secret -n <affectedClusterNamespace> \
-o yaml <affectedClusterName>-kubeconfig | awk '/admin.conf/ {print $2}' | \
head -1 | base64 -d > clusterKubeconfig.yaml

If the related cluster is regional, replace <pathToMgmtKubeconfig> with <pathToRegionalKubeconfig>.

[27192] Failure to accept new connections by ‘portforward-controller’¶

Fixed in 11.6.0 and 12.7.0

During bootstrap of a management or regional cluster of any type, portforward-controller ends accepting new connections after receiving the Accept error: “EOF” error. Hence, nothing is copied between clients.

The workaround below applies only if machines are stuck in the Provision state. Otherwise, contact Mirantis support to further assess the issue.

Workaround:

Verify that machines are stuck in the Provision state for up to 20 minutes or more. For example:
```
kubectl --kubeconfig <kindKubeconfigPath> get machines -o wide
```

Verify whether the portforward-controller Pod logs contain {{Accept error: “EOF”}} and {{Stopped forwarding}}:

kubectl --kubeconfig <kindKubeconfigPath> -n kaas logs -lapp.kubernetes.io/name=portforward-controller | grep 'Accept error: "EOF"'

kubectl --kubeconfig <kindKubeconfigPath> -n kaas logs -lapp.kubernetes.io/name=portforward-controller | grep 'Stopped forwarding'

Select from the following options:
- If the errors mentioned in the previous step are present:
 1. Restart the portforward-controller Deployment:
 kubectl --kubeconfig <kindKubeconfigPath> -n kaas rollout restart deploy portforward-controller
 2. Monitor the states of machines and the portforward-controller Pod logs. If the errors recur, restart the portforward-controller Deployment again.
- If the errors mentioned in the previous step are not present, contact Mirantis support to further assess the issue.

StackLight¶

[29329] Recreation of the Patroni container replica is stuck¶

Fixed in 11.7.0 and 12.7.0

The sample logs of the affected cluster:

INFO: doing crash recovery in a single user mode
ERROR: Crash recovery finished with code=-6
INFO:  stdout=
INFO:  stderr=2023-01-11 10:20:34 GMT [64]: [1-1] 63be8d72.40 0     LOG:  database system was interrupted; last known up at 2023-01-10 17:00:59 GMT
[64]: [2-1] 63be8d72.40 0  LOG:  could not read from log segment 00000002000000000000000F, offset 0: read 0 of 8192
[64]: [3-1] 63be8d72.40 0  LOG:  invalid primary checkpoint record
[64]: [4-1] 63be8d72.40 0  PANIC:  could not locate a valid checkpoint record

Workaround:

For the affected replica and PVC, run:

kubectl delete persistentvolumeclaim/storage-volume-patroni-<replica-id> -n stacklight

kubectl delete pod/patroni-<replica-id> -n stacklight

[28526] CPU throttling for ‘kaas-exporter’ blocking metric collection¶

Fixed in 11.6.0 and 12.7.0

A low CPU limit 100m for kaas-exporter blocks metric collection.

As a workaround, increase the CPU limit for kaas-exporter to 500m on the management cluster in the spec:providerSpec:value:kaas:management:helmReleases: section as described in {{ mos_name_abbr }} documentation: Underlay Kubernetes operations - Increase memory limits for cluster components.

[28479] Increase of the ‘metric-collector’ Pod restarts due to OOM¶

Fixed in 11.7.0 and 12.7.0

Such behavior is expected. Therefore, disregard these restarts.

[28134] Failure to update a cluster with nodes in the ‘Prepare’ state¶

Fixed in 11.6.0 and 12.7.0

A Container Cloud cluster of any type fails to update with nodes being stuck in the Prepare state and the following example error in Conditions of the affected machine:

Error: error when evicting pods/"patroni-13-2" -n "stacklight": global timeout reached: 10m0s

Other symptoms of the issue are as follows:

One of the Patroni Pods has 2/3 of containers ready. For example:

kubectl get po -n stacklight -l app=patroni

NAME           READY   STATUS    RESTARTS   AGE
patroni-13-0   3/3     Running   0          32h
patroni-13-1   3/3     Running   0          38h
patroni-13-2   2/3     Running   0          38h

The patroni-patroni-exporter container from the affected Pod is not ready. For example:

kubectl get pod/patroni-13-2 -n stacklight -o jsonpath='{.status.containerStatuses[?(@.name=="patroni-patroni-exporter")].ready}'

false

As a workaround, restart the patroni-patroni-exporter container of the affected Patroni Pod:

kubectl exec <affectedPatroniPodName> -n stacklight -c patroni-patroni-exporter -- kill 1

For example:

kubectl exec patroni-13-2 -n stacklight -c patroni-patroni-exporter -- kill 1

[27732-1] OpenSearch PVC size custom settings are dismissed during deployment¶

Fixed in 11.6.0 and 12.7.0

The OpenSearch elasticsearch.persistentVolumeClaimSize custom setting is overwritten by logging.persistentVolumeClaimSize during deployment of a Container Cloud cluster of any type and is set to the default 30Gi.

Note

This issue does not block the OpenSearch cluster operations if the default retention time is set. The default setting is usually enough for the capacity size of this cluster.

The issue may affect the following Cluster releases:

11.2.0 - 11.5.0
7.8.0 - 7.11.0
8.8.0 - 8.10.0, 12.5.0 (MOSK clusters)
10.2.4 - 10.8.1 (attached MKE 3.4.x clusters)
13.0.2 - 13.5.1 (attached MKE 3.5.x clusters)

To verify that the cluster is affected:

Note

In the commands below, substitute parameters enclosed in angle brackets to match the affected cluster values.

kubectl --kubeconfig=<managementClusterKubeconfigPath> \
-n <affectedClusterProjectName> \
get cluster <affectedClusterName> \
-o=jsonpath='{.spec.providerSpec.value.helmReleases[*].values.elasticsearch.persistentVolumeClaimSize}' | xargs echo config size:


kubectl --kubeconfig=<affectedClusterKubeconfigPath> \
-n stacklight get pvc -l 'app=opensearch-master' \
-o=jsonpath="{.items[*].status.capacity.storage}" | xargs echo capacity sizes:

The cluster is not affected if the configuration size value matches or is less than any capacity size. For example:

config size: 30Gi
capacity sizes: 30Gi 30Gi 30Gi

config size: 50Gi
capacity sizes: 100Gi 100Gi 100Gi

The cluster is affected if the configuration size is larger than any capacity size. For example:
```
config size: 200Gi
capacity sizes: 100Gi 100Gi 100Gi
```

Workaround for a new cluster creation:

Select from the following options:
- For a management or regional cluster, during the bootstrap procedure, open cluster.yaml.template for editing.
- For a managed cluster, open the Cluster object for editing.
  
  Caution
  
  For a managed cluster, use the Container Cloud API instead of the web UI for cluster creation.

In the opened .yaml file, add logging.persistentVolumeClaimSize along with elasticsearch.persistentVolumeClaimSize. For example:

apiVersion: cluster.k8s.io/v1alpha1
spec:
...
  providerSpec:
    value:
    ...
      helmReleases:
      - name: stacklight
        values:
          elasticsearch:
            persistentVolumeClaimSize: 100Gi
          logging:
            enabled: true
            persistentVolumeClaimSize: 100Gi

Continue the cluster deployment. The system will use the custom value set in logging.persistentVolumeClaimSize.

Caution

If elasticsearch.persistentVolumeClaimSize is absent in the .yaml file, the Admission Controller blocks the configuration update.

Workaround for an existing cluster:

Caution

During the application of the below workarounds, a short outage of OpenSearch and its dependent components may occur with the following alerts firing on the cluster. This behavior is expected. Therefore, disregard these alerts.

StackLight alerts list firing during cluster update

Cluster size and outage probability level	Alert name	Label name and component
Any cluster with high probability	`KubeStatefulSetOutage`	`statefulset=opensearch-master`
	`KubeDeploymentOutage`	`deployment=opensearch-dashboards` `deployment=metricbeat`
Large cluster with average probability	`KubePodsNotReady` ^{Removed in 17.0.0, 16.0.0, and 14.1.0}	`created_by_name="opensearch-master"` `created_by_name="opensearch-dashboards"` `created_by_name="metricbeat-*"`
	`OpenSearchClusterStatusWarning`	n/a
	`OpenSearchNumberOfPendingTasks`	n/a
	`OpenSearchNumberOfInitializingShards`	n/a
	`OpenSearchNumberOfUnassignedShards` ^{Removed in 2.27.0 (17.2.0 and 16.2.0)}	n/a
Any cluster with low probability	`KubeStatefulSetReplicasMismatch`	`statefulset=opensearch-master`
	`KubeDeploymentReplicasMismatch`	`deployment=opensearch-dashboards` `deployment=metricbeat`

StackLight in non-HA mode with a non-expandable StorageClass and no LVP for OpenSearch PVCs

Warning

After applying this workaround, the existing log data will be lost. Depending on your custom provisioner, you may find a third-party tool, such as pv-migrate, that provides a possibility to copy all data from one PV to another.

If data loss is acceptable, proceed with the workaround below.

Note

To verify whether a StorageClass is expandable:

kubectl -n stacklight get pvc | grep opensearch-master | awk '{print $6}' | xargs -I{} kubectl get storageclass {} -o yaml | grep 'allowVolumeExpansion: true'

A positive system response is allowVolumeExpansion: true. A negative system response is blank or false.

Scale down the opensearch-master StatefulSet with dependent resources to 0 and disable the elasticsearch-curator CronJob:

kubectl -n stacklight scale --replicas 0 statefulset opensearch-master

kubectl -n stacklight scale --replicas 0 deployment opensearch-dashboards

kubectl -n stacklight scale --replicas 0 deployment metricbeat

kubectl -n stacklight patch cronjobs elasticsearch-curator -p '{"spec" : {"suspend" : true }}'

Recreate the opensearch-master StatefulSet with the updated disk size:

kubectl get statefulset opensearch-master -o yaml -n stacklight | sed 's/storage: 30Gi/storage: <<pvc_size>>/g' > opensearch-master.yaml

kubectl -n stacklight delete statefulset opensearch-master

kubectl create -f opensearch-master.yaml

Replace <pvcSize> with the elasticsearch.persistentVolumeClaimSize value.

Delete existing PVCs:
```
kubectl delete pvc -l 'app=opensearch-master' -n stacklight
```
Warning

This command removes all existing logs data from PVCs.

In the Cluster configuration, set logging.persistentVolumeClaimSize to the same value as the size of the elasticsearch.persistentVolumeClaimSize parameter. For example:

 apiVersion: cluster.k8s.io/v1alpha1
 kind: Cluster
 spec:
 ...
   providerSpec:
     value:
     ...
       helmReleases:
       - name: stacklight
         values:
           elasticsearch:
             persistentVolumeClaimSize: 100Gi
           logging:
             enabled: true
             persistentVolumeClaimSize: 100Gi

Scale up the opensearch-master StatefulSet with dependent resources to 1 and enable the elasticsearch-curator CronJob:

kubectl -n stacklight scale --replicas 1 statefulset opensearch-master

sleep 100

kubectl -n stacklight scale --replicas 1 deployment opensearch-dashboards

kubectl -n stacklight scale --replicas 1 deployment metricbeat

kubectl -n stacklight patch cronjobs elasticsearch-curator -p '{"spec" : {"suspend" : false }}'

[27732-2] Custom settings for ‘elasticsearch.logstashRetentionTime’ are dismissed¶

Fixed in 11.6.0 and 12.7.0

Custom settings for the deprecated elasticsearch.logstashRetentionTime parameter are overwritten by the default setting set to 1 day.

The issue may affect the following Cluster releases with enabled elasticsearch.logstashRetentionTime:

11.2.0 - 11.5.0
7.8.0 - 7.11.0
8.8.0 - 8.10.0, 12.5.0 (MOSK clusters)
10.2.4 - 10.8.1 (attached MKE 3.4.x clusters)
13.0.2 - 13.5.1 (attached MKE 3.5.x clusters)

As a workaround, in the Cluster object, replace elasticsearch.logstashRetentionTime with elasticsearch.retentionTime that was implemented to replace the deprecated parameter. For example:

apiVersion: cluster.k8s.io/v1alpha1
kind: Cluster
spec:
  ...
  providerSpec:
    value:
    ...
      helmReleases:
      - name: stacklight
        values:
          elasticsearch:
            retentionTime:
              logstash: 10
              events: 10
              notifications: 10
          logging:
            enabled: true

For the StackLight configuration procedure and parameters description, refer to Configure StackLight.

[20876] StackLight pods get stuck with the ‘NodeAffinity failed’ error¶

Note

Moving forward, the workaround for this issue will be moved from Release Notes to Operations Guide: Troubleshoot StackLight.

The issue does not affect the StackLight services, all required StackLight pods migrate successfully except extra pods that are created and stuck during pod migration.

As a workaround, remove the stuck pods:

kubectl --kubeconfig <managedClusterKubeconfig> -n stacklight delete pod <stuckPodName>

Storage¶

[28783] Ceph conditon stuck in absence of Ceph cluster secrets info¶

Fixed in 11.6.0 and 12.7.0

Ceph conditon gets stuck in absence of the Ceph cluster secrets information. The observed behaviour can be found on the MOSK 22.3 clusters running on top of Container Cloud 2.21.

The list of the symptoms includes:

The Cluster object contains the following condition:

Failed to configure Ceph cluster: ceph cluster status info is not \
updated at least for 5 minutes, ceph cluster secrets info is not available yet

The ceph-kcc-controller logs from the kaas namespace contain the following loglines:

2022-11-30 19:39:17.393595 E | ceph-spec: failed to update cluster condition to \
{Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully \
LastHeartbeatTime:2022-11-30 19:39:17.378401993 +0000 UTC m=+2617.717554955 \
LastTransitionTime:2022-05-16 16:14:37 +0000 UTC}. failed to update object \
"rook-ceph/rook-ceph" status: Operation cannot be fulfilled on \
cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please \
apply your changes to the latest version and try again

Workaround:

Edit KaaSCephCluster of the affected managed cluster:
```
kubectl -n <managedClusterProject> edit kaascephcluster
```
Substitute <managedClusterProject> with the corresponding managed cluster namespace.
Define the version parameter in the KaaSCephCluster spec:
```
spec:
  cephClusterSpec:
    version: 15.2.13
```
Note

Starting from MOSK 22.4, the Ceph cluster version updates to 15.2.17. Therefore, remove the version parameter definition from KaaSCephCluster after the managed cluster update.

Save the updated KaaSCephCluster spec.

Find the MiraCeph Custom Resource on a managed cluster and copy all annotations starting with meta.helm.sh:

kubectl --kubeconfig <managedClusterKubeconfig> get crd miracephs.lcm.mirantis.com -o yaml

Substitute <managedClusterKubeconfig> with a corresponding managed cluster kubeconfig.

Example of a system output:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.0
    # save all annotations with "meta.helm.sh" somewhere
    meta.helm.sh/release-name: ceph-controller
    meta.helm.sh/release-namespace: ceph
...

Create the miracephsecretscrd.yaml file and fill it with the following template:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.0
    <insert all "meta.helm.sh" annotations here>
  labels:
    app.kubernetes.io/managed-by: Helm
  name: miracephsecrets.lcm.mirantis.com
spec:
  conversion:
    strategy: None
  group: lcm.mirantis.com
  names:
    kind: MiraCephSecret
    listKind: MiraCephSecretList
    plural: miracephsecrets
    singular: miracephsecret
  scope: Namespaced
  versions:
    - name: v1alpha1
      schema:
        openAPIV3Schema:
          description: MiraCephSecret aggregates secrets created by Ceph
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            status:
              properties:
                lastSecretCheck:
                  type: string
                lastSecretUpdate:
                  type: string
                messages:
                  items:
                    type: string
                  type: array
                state:
                  type: string
              type: object
          type: object
      served: true
      storage: true

Insert the copied meta.helm.sh annotations to the metadata.annotations section of the template.

Apply miracephsecretscrd.yaml on the managed cluster:
```
kubectl --kubeconfig <managedClusterKubeconfig> apply -f miracephsecretscrd.yaml
```
Substitute <managedClusterKubeconfig> with a corresponding managed cluster kubeconfig.
Obtain the MiraCeph name from the managed cluster:
```
kubectl --kubeconfig <managedClusterKubeconfig> -n ceph-lcm-mirantis get miraceph -o name
```
Substitute <managedClusterKubeconfig> with the corresponding managed cluster kubeconfig.

Example of a system output:
```
miraceph.lcm.mirantis.com/rook-ceph
```
Copy the MiraCeph name after slash, the rook-ceph part from the example above.

Create the mcs.yaml file and fill it with the following template:

apiVersion: lcm.mirantis.com/v1alpha1
kind: MiraCephSecret
metadata:
  name: <miracephName>
  namespace: ceph-lcm-mirantis
status: {}

Substitute <miracephName> with the MiraCeph name from the previous step.

Apply mcs.yaml on the managed cluster:
```
kubectl --kubeconfig <managedClusterKubeconfig> apply -f mcs.yaml
```
Substitute <managedClusterKubeconfig> with a corresponding managed cluster kubeconfig.

After some delay, the cluster condition will be updated to the health state.

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

Components versions¶

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.21.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

**Container Cloud release components versions**¶
Component	Application/Service	Version
AWS ^Updated	aws-provider	1.34.16
	aws-credentials-controller	1.34.16
Azure ^Updated	azure-provider	1.34.16
	azure-credentials-controller	1.34.16
Bare metal	ambassador	1.20.1-alpine
	baremetal-operator ^Updated	base-focal-20220611131433
	baremetal-public-api ^Updated	1.34.16
	baremetal-provider ^Updated	1.34.16
	baremetal-resource-controller	base-focal-20220627134752
	ironic	yoga-focal-20220719132049
	kaas-ipam	base-focal-20220503165133
	keepalived ^Updated	0.19.0-5-g6a7e17d
	local-volume-provisioner ^Updated	2.4.0
	mariadb	10.4.17-bionic-20220113085105
	metallb-controller ^Updated	0.13.4 0
IAM	iam ^Updated	2.4.35
	iam-controller ^Updated	1.34.16
	keycloak	18.0.0
Container Cloud ^Updated	admission-controller	1.34.16
	agent-controller	1.34.16
	byo-credentials-controller	1.34.16
	byo-provider	1.34.16
	ceph-kcc-controller	1.34.16
	cert-manager	1.34.16
	client-certificate-controller	1.34.16
	event-controller	1.34.16
	golang	1.18.5
	kaas-public-api	1.34.16
	kaas-exporter	1.34.16
	kaas-ui	1.34.16
	license-controller	1.34.16
	lcm-controller	0.3.0-327-gbc30b11b
	machinepool-controller	1.34.16
	mcc-cache	1.34.16
	metrics-server	0.5.2
	portforward-controller	1.34.16
	proxy-controller	1.34.16
	rbac-controller	1.34.16
	release-controller	1.34.16
	rhellicense-controller	1.34.16
	scope-controller	1.34.16
	user-controller	1.34.16
Equinix Metal ^Updated	equinix-provider	1.34.16
	equinix-credentials-controller	1.34.16
	keepalived	0.19.0-5-g6a7e17d
OpenStack ^Updated	openstack-provider	1.34.16
	os-credentials-controller	1.34.16
VMware vSphere ^Updated	metallb-controller ^Updated	0.13.4
	vsphere-provider	1.34.16
	vsphere-credentials-controller	1.34.16
	keepalived	0.19.0-5-g6a7e17d
	squid-proxy	0.0.1-7

0: For MOSK-based deployments, the metallb-controller version is updated from 0.12.1 to 0.13.4 in MOSK 22.5.

Artifacts¶

This section lists the components artifacts of the Mirantis Container Cloud release 2.21.0.

Bare metal artifacts
Core artifacts
IAM artifacts

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶
Artifact	Component	Path
Binaries	baremetal-api ^Updated	https://binary.mirantis.com/core/helm/baremetal-api-1.34.16.tgz
	baremetal-operator ^Updated	https://binary.mirantis.com/core/helm/baremetal-operator-1.34.17.tgz
	baremetal-public-api ^Updated	https://binary.mirantis.com/core/helm/baremetal-public-api-1.34.16.tgz
	ironic-python-agent.initramfs ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20220915111547
	ironic-python-agent.kernel ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20220915111547
	kaas-ipam ^Updated	https://binary.mirantis.com/core/helm/kaas-ipam-1.34.16.tgz
	metallb	https://binary.mirantis.com/core/helm/metallb-1.34.16.tgz
	local-volume-provisioner ^Updated	https://binary.mirantis.com/core/helm/local-volume-provisioner-1.34.16.tgz
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-104-6e2e82c.tgz
Docker images	ambassador	mirantis.azurecr.io/general/external/docker.io/library/nginx:1.20.1-alpine
	baremetal-operator	mirantis.azurecr.io/bm/baremetal-operator:base-focal-20220611131433
	baremetal-resource-controller	mirantis.azurecr.io/bm/baremetal-resource-controller:base-focal-20220627134752
	dynamic_ipxe ^Updated	mirantis.azurecr.io/bm/dynamic-ipxe:base-focal-20221018205745
	dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-alpine-20221025105458
	dnsmasq-controller ^Updated	mirantis.azurecr.io/bm/dnsmasq-controller:base-focal-20220811133223
	ironic	mirantis.azurecr.io/openstack/ironic:yoga-focal-20220719132049
	ironic-inspector	mirantis.azurecr.io/openstack/ironic-inspector:yoga-focal-20220719132049
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20220602121226
	kaas-ipam	mirantis.azurecr.io/bm/kaas-ipam:base-focal-20220503165133
	mariadb	mirantis.azurecr.io/general/mariadb:10.4.17-bionic-20220113085105
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.19.0-5-g6a7e17d
	metallb-controller ^Updated 0	mirantis.azurecr.io/bm/external/metallb/controller:v0.13.4
	metallb-speaker ^Updated 0	mirantis.azurecr.io/bm/external/metallb/speaker:v0.13.4
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-focal-20220128103433

0(1,2): For MOSK-based deployments, the metallb version is updated from 0.12.1 to 0.13.4 in MOSK 22.5.

Core artifacts¶
Artifact	Component	Paths
Bootstrap tarball ^Updated	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.34.16.tar.gz
	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.34.16.tar.gz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.34.16.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.34.16.tgz
	aws-credentials-controller	https://binary.mirantis.com/core/helm/aws-credentials-controller-1.34.16.tgz
	aws-provider	https://binary.mirantis.com/core/helm/aws-provider-1.34.16.tgz
	azure-credentials-controller	https://binary.mirantis.com/core/helm/azure-credentials-controller-1.34.16.tgz
	azure-provider	https://binary.mirantis.com/core/helm/azure-provider-1.34.16.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.34.16.tgz
	byo-credentials-controller	https://binary.mirantis.com/core/helm/byo-credentials-controller-1.34.16.tgz
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.34.16.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.34.16.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.34.16.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.34.16.tgz
	configuration-collector ^New	https://binary.mirantis.com/core/helm/configuration-collector-1.34.16.tgz
	equinix-credentials-controller	https://binary.mirantis.com/core/helm/equinix-credentials-controller-1.34.16.tgz
	equinix-provider	https://binary.mirantis.com/core/helm/equinix-provider-1.34.16.tgz
	equinixmetalv2-provider	https://binary.mirantis.com/core/helm/equinixmetalv2-provider-1.34.16.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.34.16.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.34.16.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.34.16.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.34.16.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.34.16.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.34.16.tgz
	license-controller	https://binary.mirantis.com/core/helm/license-controller-1.34.16.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.34.16.tgz
	metrics-server	https://binary.mirantis.com/core/helm/metrics-server-1.34.16.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.34.16.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.34.16.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.34.16.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.34.16.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.34.16.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.34.16.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.34.16.tgz
	scope-controller	http://binary.mirantis.com/core/helm/scope-controller-1.34.16.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.34.16.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.34.16.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.34.16.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.34.16.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.34.16
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.34.16
	aws-cluster-api-controller ^Updated	mirantis.azurecr.io/core/aws-cluster-api-controller:1.34.16
	aws-credentials-controller ^Updated	mirantis.azurecr.io/core/aws-credentials-controller:1.34.16
	azure-cluster-api-controller ^Updated	mirantis.azurecr.io/core/azure-cluster-api-controller:1.34.16
	azure-credentials-controller ^Updated	mirantis.azurecr.io/core/azure-credentials-controller:1.34.16
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.34.16
	byo-credentials-controller ^Updated	mirantis.azurecr.io/core/byo-credentials-controller:1.34.16
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.34.16
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.6.1
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.34.16
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.34.16
	cluster-api-provider-equinix ^Updated	mirantis.azurecr.io/core/equinix-cluster-api-controller:1.34.16
	equinix-credentials-controller ^Updated	mirantis.azurecr.io/core/equinix-credentials-controller:1.34.16
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.34.16
	haproxy	mirantis.azurecr.io/lcm/mcc-haproxy:v0.17.0-8-g6ca89d5
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.34.16
	kaas-exporter	mirantis.azurecr.io/core/kaas-exporter:1.34.16
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.34.16
	lcm-controller ^Updated	mirantis.azurecr.io/lcm/lcm-controller:v0.3.0-327-gbc30b11b
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.34.16
	machinepool-controller ^Updated	mirantis.azurecr.io/core/machinepool-controller:1.34.16
	mcc-keepalived ^Updated	mirantis.azurecr.io/lcm/mcc-keepalived:v0.19.0-5-g6a7e17d
	metrics-server	mirantis.azurecr.io/core/external/metrics-server:v0.5.2
	nginx	mirantis.azurecr.io/core/external/nginx:1.34.16
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.34.16
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.34.16
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.34.16
	proxy-controller ^Updated	mirantis.azurecr.io/core/proxy-controller:1.34.16
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.34.16
	registry	mirantis.azurecr.io/lcm/registry:2.7.1
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.34.16
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.34.16
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.34.16
	squid-proxy ^Updated	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-7
	storage-discovery	mirantis.azurecr.io/core/storage-discovery:1.34.16
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.34.16
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.34.16
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.34.16

IAM artifacts¶
Artifact	Component	Path
Binaries	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
Helm charts ^Updated	iam	http://binary.mirantis.com/iam/helm/iam-2.4.35.tgz
	iam-proxy	http://binary.mirantis.com/iam/helm/iam-proxy-0.2.13.tgz
	keycloak_proxy	http://binary.mirantis.com/core/helm/keycloak_proxy-1.34.16.tgz
Docker images	kubernetes-entrypoint	mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.0-20200311160233
	mariadb ^Updated	mirantis.azurecr.io/general/mariadb:10.6.7-focal-20220811085105
	keycloak ^Updated	mirantis.azurecr.io/iam/keycloak:0.5.12
	keycloak-gatekeeper ^Updated	mirantis.azurecr.io/iam/keycloak-gatekeeper:7.1.3-3

Post-upgrade actions¶

Since Kubernetes policy does not allow updating images in existing IAM jobs, after Container Cloud automatically upgrades to 2.21.0, update the MariaDB image manually using the following steps:

Delete the existing job:

kubectl delete job -n kaas iam-cluster-wait

In the management Cluster object, and add following snippet:

kaas:
  management:
    enabled: true
    helmReleases:
    - name: iam
      values:
        keycloak:
          mariadb:
            images:
              tags:
                mariadb_scripted_test: general/mariadb:10.6.7-focal-20220811085105

Wait until helm-controller applies changes.

Verify that the job was recreated and the new image was added:

kubectl describe job -n kaas iam-cluster-wait | grep -i image

See also

2.20.1¶

The Mirantis Container Cloud GA release 2.20.1 is based on 2.20.0 and:

Introduces support for the Cluster release 8.10.0 that is based on the Cluster release 7.10.0 and represents Mirantis OpenStack for Kubernetes (MOSK) 22.4.

This Cluster release is based on the updated version of Mirantis Kubernetes Engine 3.4.10 with Kubernetes 1.20 and Mirantis Container Runtime 20.10.12.
Supports the latest Cluster releases 7.10.0 and 11.4.0.
Does not support greenfield deployments based on deprecated Cluster releases 11.3.0, 8.8.0, and 7.9.0. Use the latest available Cluster releases of the series instead.

For details about the Container Cloud release 2.20.1, refer to its parent release 2.20.0:

Caution

See also

2.20.0¶

The Mirantis Container Cloud GA release 2.20.0:

Introduces support for the Cluster release 11.4.0 that is based on Mirantis Container Runtime 20.10.12 and Mirantis Kubernetes Engine 3.5.4 with Kubernetes 1.21.
Introduces support for the Cluster release 7.10.0 that is based on Mirantis Container Runtime 20.10.12 and Mirantis Kubernetes Engine 3.4.10 with Kubernetes 1.20.
Supports the Cluster release 8.8.0 that is based on the Cluster release 7.8.0 and represents Mirantis OpenStack for Kubernetes (MOSK) 22.3.
Does not support greenfield deployments on deprecated Cluster releases 11.3.0, 8.6.0, and 7.9.0. Use the latest available Cluster releases of the series instead.

Caution

Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.20.0.

Enhancements¶

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.20.0. For the list of enhancements in the Cluster releases 11.4.0 and 7.10.0 that are introduced by the Container Cloud release 2.20.0, see the Cluster releases (managed).

IAM ‘member’ role
Bastion node configuration for OpenStack and AWS manged clusters
Mandatory IPAM service label for bare metal LCM subnets
Flexible size units for bare metal host profiles
General availability support for MITM proxy
Configuration of TLS certificates for ‘mcc-cache’ and MKE
Documentation enhancements

IAM ‘member’ role¶

Added the IAM member role to the existing IAM roles list. The Infrastructure Operator with the member role has the read and write access to Container Cloud API allowing cluster operations and does not have access to IAM objects.

Learn more

Operations Guide: Available IAM roles and use cases

Bastion node configuration for OpenStack and AWS manged clusters¶

Implemented the capability to configure the Bastion node on greenfield deployments of the OpenStack-based and AWS-based managed clusters using the Container Cloud web UI. Using the Create Cluster wizard, you can now configure the following parameters for the Bastion node:

OpenStack-based: flavor, image, availability zone, server metadata, booting from a volume
AWS-based: instance type, AMI ID

Note

Reconfiguration of the Bastion node on an existing cluster is not supported.

Mandatory IPAM service label for bare metal LCM subnets¶

Made the ipam/SVC-k8s-lcm label mandatory for the LCM subnet on new deployments of management and managed bare metal clusters. It allows the LCM Agent to correctly identify IP addresses to use on multi-homed bare metal hosts. Therefore, you must add this label explicitly on new clusters.

Each node of every cluster must now have only one IP address in the LCM network that is allocated from one of the Subnet objects having the ipam/SVC-k8s-lcm label defined. Therefore, all Subnet objects used for LCM networks must have the ipam/SVC-k8s-lcm label defined.

Note

For MOSK-based deployments, the feature support is available since MOSK 22.4.

Learn more

Flexible size units for bare metal host profiles¶

Implemented the possibility to use flexible size units throughout bare metal host profiles for management, regional, and managed clusters. For example, you can now use either sizeGiB: 0.1 or size: 100Mi when specifying a device size. The size without units is counted in bytes. For example, size: 120 means 120 bytes.

Note

For MOSK-based deployments, the feature support is available since MOSK 22.4.

Learn more

General availability support for MITM proxy¶

Completed integration of the man-in-the-middle (MITM) proxies support for offline deployments by adding AWS, vSphere, and Equinix Metal with private networking to the list of existing supported providers: OpenStack and bare metal.

With trusted proxy CA certificates that you can now add using the CA Certificate check box in the Add new Proxy window during a managed cluster creation, the feature allows monitoring all cluster traffic for security and audit purposes.

Note

For Azure and Equinix Metal with public networking, the feature is not supported
For MOSK-based deployments, the feature support will become available in one of the following Container Cloud releases.

Learn more

Configuration of TLS certificates for ‘mcc-cache’ and MKE¶

Implemented the ability to configure TLS certificates for mcc-cache on management or regional clusters and for MKE on managed clusters deployed or updated by Container Cloud using the latest Cluster release.

Note

TLS certificates configuration for MKE is not supported:

For MOSK-based clusters
For attached MKE clusters that were not originally deployed by Container Cloud

Learn more

Operations Guide: Configure TLS certificates for cluster applications

Documentation enhancements¶

On top of continuous improvements delivered to the existing Container Cloud guides, added a document on how to increase the overall storage size for all Ceph pools of the same device class: hdd, ssd, or nvme. For details, see Increase Ceph cluster storage size.

Addressed issues¶

The following issues have been addressed in the Mirantis Container Cloud release 2.20.0 along with the Cluster releases 11.4.0 and 7.10.0:

[25476] Fixed the timeout behavior to avoid Keepalived and HAProxy check failures.
[25076] Fixed the remote_syslog configuration. Now, you can optionally define SSL verification modes. For details, see MOSK Operations Guide: StackLight configuration parameters - Logging to syslog.
[24927] Fixed the issue wherein a failure to create lcmclusterstate did not trigger a retry.
[24852] Fixed the issue wherein the Upgrade Schedule tab in the Container Cloud web UI was displaying the NOT ALLOWED label instead of ALLOWED if the upgrade was enabled.
[24837] Fixed the issue wherein some Keycloak iam-keycloak-* pods were in the CrashLoopBackOff state during an update of a baremetal-based management or managed cluster with enabled FIPs.
[24813] Fixed the issue wherein the IPaddr objects were not reconciled after the ipam/SVC-* labels changed on the parent subnet. This prevented the ipam/SVC-* labels from propagating to IPaddr objects and caused the serviceMap update to fail in the corresponding IpamHost.
[23125] Fixed the issue wherein an OpenStack-based regional cluster creation in an offline mode was failing. Adding the Kubernetes load balancer address to the NO_PROXY environment variable is no longer required.
[22576] Fixed the issue wherein provisioning-ansible did not use the wipe flags during the deployment phase.
[5238] Improved the Bastion readiness checks to avoid issues with some clusters having several Bastion nodes.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.20.0 including the Cluster releases 11.4.0 and 7.10.0.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

MKE
Bare metal
Equinix Metal with private networking
vSphere

StackLight
Ceph
Management cluster upgrade
Container Cloud web UI

MKE¶

[20651] A cluster deployment or update fails with not ready compose deployments¶

'not ready: deployments: kube-system/compose got 0/0 replicas, kube-system/compose-api
 got 0/0 replicas'
 ready: false
 type: Kubernetes

Workaround:

Disable Docker Content Trust (DCT):
1. Access the MKE web UI as admin.
2. Navigate to Admin > Admin Settings.
3. In the left navigation pane, click Docker Content Trust and disable it.
Restart the affected deployments such as calico-kube-controllers, compose, compose-api, coredns, and so on:
```
kubectl -n kube-system delete deployment <deploymentName>
```
Once done, the cluster deployment or update resumes.
Re-enable DCT.

Bare metal¶

[26659] Regional cluster deployment failure with stuck ‘mcc-cache’ Pods¶

Fixed in 11.6.0

Deployment of a regional cluster based on bare metal or Equinix Metal with private networking fails with mcc-cache Pods being stuck in the CrashLoopBackOff status of restarts.

As a workaround, remove failed mcc-cache Pods to restart them automatically. For example:

kubectl -n kaas delete pod mcc-cache-0

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

[20736] Region deletion failure after regional deployment failure¶

If a baremetal-based regional cluster deployment fails before pivoting is done, the corresponding region deletion fails.

Workaround:

cluster
machine
baremetalhost
baremetalhostprofile
l2template
subnet
ipamhost
ipaddr

kubectl delete <objectName> -l kaas.mirantis.com/region=<regionName>

Warning

Do not use the same region name again after the regional cluster deployment failure since some objects that reference the region name may still exist.

Equinix Metal with private networking¶

[26659] Regional cluster deployment failure with stuck ‘mcc-cache’ Pods¶

Fixed in 11.6.0

Deployment of a regional cluster based on bare metal or Equinix Metal with private networking fails with mcc-cache Pods being stuck in the CrashLoopBackOff status of restarts.

As a workaround, remove failed mcc-cache Pods to restart them automatically. For example:

kubectl -n kaas delete pod mcc-cache-0

vSphere¶

[26070] RHEL system cannot be registered in Red Hat portal over MITM proxy¶

Unable to verify server's identity: [SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed (_ssl.c:618)

The Container Cloud deployment gets stuck while applying the RHEL license to machines with the same error in the lcm-agent logs.

As a workaround, use the internal Red Hat Satellite server that a VM can access directly without a MITM proxy.

StackLight¶

[28526] CPU throttling for ‘kaas-exporter’ blocking metric collection¶

Fixed in 11.6.0 and 12.7.0

A low CPU limit 100m for kaas-exporter blocks metric collection.

[27732-1] OpenSearch PVC size custom settings are dismissed during deployment¶

Fixed in 11.6.0 and 12.7.0

Note

This issue does not block the OpenSearch cluster operations if the default retention time is set. The default setting is usually enough for the capacity size of this cluster.

The issue may affect the following Cluster releases:

11.2.0 - 11.5.0
7.8.0 - 7.11.0
8.8.0 - 8.10.0, 12.5.0 (MOSK clusters)
10.2.4 - 10.8.1 (attached MKE 3.4.x clusters)
13.0.2 - 13.5.1 (attached MKE 3.5.x clusters)

To verify that the cluster is affected:

Note

In the commands below, substitute parameters enclosed in angle brackets to match the affected cluster values.

kubectl --kubeconfig=<managementClusterKubeconfigPath> \
-n <affectedClusterProjectName> \
get cluster <affectedClusterName> \
-o=jsonpath='{.spec.providerSpec.value.helmReleases[*].values.elasticsearch.persistentVolumeClaimSize}' | xargs echo config size:


kubectl --kubeconfig=<affectedClusterKubeconfigPath> \
-n stacklight get pvc -l 'app=opensearch-master' \
-o=jsonpath="{.items[*].status.capacity.storage}" | xargs echo capacity sizes:

The cluster is not affected if the configuration size value matches or is less than any capacity size. For example:

config size: 30Gi
capacity sizes: 30Gi 30Gi 30Gi

config size: 50Gi
capacity sizes: 100Gi 100Gi 100Gi

The cluster is affected if the configuration size is larger than any capacity size. For example:
```
config size: 200Gi
capacity sizes: 100Gi 100Gi 100Gi
```

Workaround for a new cluster creation:

Select from the following options:
- For a management or regional cluster, during the bootstrap procedure, open cluster.yaml.template for editing.
- For a managed cluster, open the Cluster object for editing.
  
  Caution
  
  For a managed cluster, use the Container Cloud API instead of the web UI for cluster creation.

In the opened .yaml file, add logging.persistentVolumeClaimSize along with elasticsearch.persistentVolumeClaimSize. For example:

apiVersion: cluster.k8s.io/v1alpha1
spec:
...
  providerSpec:
    value:
    ...
      helmReleases:
      - name: stacklight
        values:
          elasticsearch:
            persistentVolumeClaimSize: 100Gi
          logging:
            enabled: true
            persistentVolumeClaimSize: 100Gi

Continue the cluster deployment. The system will use the custom value set in logging.persistentVolumeClaimSize.

Caution

If elasticsearch.persistentVolumeClaimSize is absent in the .yaml file, the Admission Controller blocks the configuration update.

Workaround for an existing cluster:

Caution

StackLight alerts list firing during cluster update

Cluster size and outage probability level	Alert name	Label name and component
Any cluster with high probability	`KubeStatefulSetOutage`	`statefulset=opensearch-master`
	`KubeDeploymentOutage`	`deployment=opensearch-dashboards` `deployment=metricbeat`
Large cluster with average probability	`KubePodsNotReady` ^{Removed in 17.0.0, 16.0.0, and 14.1.0}	`created_by_name="opensearch-master"` `created_by_name="opensearch-dashboards"` `created_by_name="metricbeat-*"`
	`OpenSearchClusterStatusWarning`	n/a
	`OpenSearchNumberOfPendingTasks`	n/a
	`OpenSearchNumberOfInitializingShards`	n/a
	`OpenSearchNumberOfUnassignedShards` ^{Removed in 2.27.0 (17.2.0 and 16.2.0)}	n/a
Any cluster with low probability	`KubeStatefulSetReplicasMismatch`	`statefulset=opensearch-master`
	`KubeDeploymentReplicasMismatch`	`deployment=opensearch-dashboards` `deployment=metricbeat`

StackLight in non-HA mode with a non-expandable StorageClass and no LVP for OpenSearch PVCs

Warning

If data loss is acceptable, proceed with the workaround below.

Note

To verify whether a StorageClass is expandable:

kubectl -n stacklight get pvc | grep opensearch-master | awk '{print $6}' | xargs -I{} kubectl get storageclass {} -o yaml | grep 'allowVolumeExpansion: true'

A positive system response is allowVolumeExpansion: true. A negative system response is blank or false.

Scale down the opensearch-master StatefulSet with dependent resources to 0 and disable the elasticsearch-curator CronJob:

kubectl -n stacklight scale --replicas 0 statefulset opensearch-master

kubectl -n stacklight scale --replicas 0 deployment opensearch-dashboards

kubectl -n stacklight scale --replicas 0 deployment metricbeat

kubectl -n stacklight patch cronjobs elasticsearch-curator -p '{"spec" : {"suspend" : true }}'

Recreate the opensearch-master StatefulSet with the updated disk size:

kubectl get statefulset opensearch-master -o yaml -n stacklight | sed 's/storage: 30Gi/storage: <<pvc_size>>/g' > opensearch-master.yaml

kubectl -n stacklight delete statefulset opensearch-master

kubectl create -f opensearch-master.yaml

Replace <pvcSize> with the elasticsearch.persistentVolumeClaimSize value.

Delete existing PVCs:
```
kubectl delete pvc -l 'app=opensearch-master' -n stacklight
```
Warning

This command removes all existing logs data from PVCs.

In the Cluster configuration, set logging.persistentVolumeClaimSize to the same value as the size of the elasticsearch.persistentVolumeClaimSize parameter. For example:

 apiVersion: cluster.k8s.io/v1alpha1
 kind: Cluster
 spec:
 ...
   providerSpec:
     value:
     ...
       helmReleases:
       - name: stacklight
         values:
           elasticsearch:
             persistentVolumeClaimSize: 100Gi
           logging:
             enabled: true
             persistentVolumeClaimSize: 100Gi

Scale up the opensearch-master StatefulSet with dependent resources to 1 and enable the elasticsearch-curator CronJob:

kubectl -n stacklight scale --replicas 1 statefulset opensearch-master

sleep 100

kubectl -n stacklight scale --replicas 1 deployment opensearch-dashboards

kubectl -n stacklight scale --replicas 1 deployment metricbeat

kubectl -n stacklight patch cronjobs elasticsearch-curator -p '{"spec" : {"suspend" : false }}'

[27732-2] Custom settings for ‘elasticsearch.logstashRetentionTime’ are dismissed¶

Fixed in 11.6.0 and 12.7.0

Custom settings for the deprecated elasticsearch.logstashRetentionTime parameter are overwritten by the default setting set to 1 day.

The issue may affect the following Cluster releases with enabled elasticsearch.logstashRetentionTime:

11.2.0 - 11.5.0
7.8.0 - 7.11.0
8.8.0 - 8.10.0, 12.5.0 (MOSK clusters)
10.2.4 - 10.8.1 (attached MKE 3.4.x clusters)
13.0.2 - 13.5.1 (attached MKE 3.5.x clusters)

As a workaround, in the Cluster object, replace elasticsearch.logstashRetentionTime with elasticsearch.retentionTime that was implemented to replace the deprecated parameter. For example:

apiVersion: cluster.k8s.io/v1alpha1
kind: Cluster
spec:
  ...
  providerSpec:
    value:
    ...
      helmReleases:
      - name: stacklight
        values:
          elasticsearch:
            retentionTime:
              logstash: 10
              events: 10
              notifications: 10
          logging:
            enabled: true

For the StackLight configuration procedure and parameters description, refer to Configure StackLight.

[20876] StackLight pods get stuck with the ‘NodeAffinity failed’ error¶

Note

Moving forward, the workaround for this issue will be moved from Release Notes to Operations Guide: Troubleshoot StackLight.

The issue does not affect the StackLight services, all required StackLight pods migrate successfully except extra pods that are created and stuck during pod migration.

As a workaround, remove the stuck pods:

kubectl --kubeconfig <managedClusterKubeconfig> -n stacklight delete pod <stuckPodName>

Ceph¶

[26820] ‘KaaSCephCluster’ does not reflect issues during Ceph cluster deletion¶

Fixed in 2.22.0

The status section in the KaaSCephCluster.status CR does not reflect issues during the process of a Ceph cluster deletion.

As a workaround, inspect Ceph Controller logs on the managed cluster:

kubectl --kubeconfig <managedClusterKubeconfig> -n ceph-lcm-mirantis logs <ceph-controller-pod-name>

[26441] Cluster update fails with the MountDevice failed for volume warning¶

Workaround:

Verify that the description of the Pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
In the command above, replace the following values:
- <affectedProjectName> is the Container Cloud project name where the Pods failed to run
- <affectedPodName> is a Pod name that failed to run in the specified project
In the Pod description, identify the node name where the Pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the Pod that fails to 0 replicas.

On every csi-rbdplugin Pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected Pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state becomes Running.

Management cluster upgrade¶

[26740] Failure to upgrade a management cluster with a custom certificate¶

Fixed in 2.21.0

An upgrade of a Container Cloud management cluster with a custom Keycloak or web UI TLS certificate fails with the following example error:

failed to update management cluster: \
admission webhook "validations.kaas.mirantis.com" denied the request: \
failed to validate TLS spec for Cluster 'default/kaas-mgmt': \
desired hostname is not set for 'ui'

Workaround:

Verify that the tls section of the management cluster contains the hostname and certificate fields for configured applications:

Open the management Cluster object for editing:
```
kubectl edit cluster <mgmtClusterName>
```

Verify that the tls section contains the following fields:

tls:
  keycloak:
    certificate:
      name: keycloak
    hostname: <keycloakHostName>
    tlsConfigRef: “” or “keycloak”
  ui:
    certificate:
      name: ui
    hostname: <webUIHostName>
    tlsConfigRef: “” or “ui”

Container Cloud web UI¶

[26416] Failure to upload an MKE client bundle during cluster attachment¶

Fixed in 7.11.0, 11.5.0 and 12.5.0

During attachment of an existing MKE cluster using the Container Cloud web UI, uploading of an MKE client bundle fails with a false-positive message about a successful uploading.

Workaround:

Select from the following options:

Fill in the required fields for the MKE client bundle manually.
In the Attach Existing MKE Cluster window, use upload MKE client bundle twice to upload ucp.bundle-admin.zip and ucp-docker-bundle.zip located in the first archive.

[23002] Inability to set a custom value for a predefined node label¶

Fixed in 7.11.0, 11.5.0 and 12.5.0

During machine creation using the Container Cloud web UI, a custom value for a node label cannot be set.

As a workaround, manually add the value to spec.providerSpec.value.nodeLabels in machine.yaml.

Components versions¶

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.20.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

**Container Cloud release components versions**¶
Component	Application/Service	Version
AWS ^Updated	aws-provider	1.33.5
	aws-credentials-controller	1.33.5
Azure ^Updated	azure-provider	1.33.5
	azure-credentials-controller	1.33.5
Bare metal	ambassador	1.20.1-alpine
	baremetal-operator ^Updated	6.3.3
	baremetal-public-api ^Updated	6.3.3
	baremetal-provider ^Updated	1.33.5
	baremetal-resource-controller ^Updated	base-focal-20220627134752
	ironic ^Updated	yoga-focal-20220719132049
	ironic-operator ^Removed	n/a
	kaas-ipam	base-focal-20220503165133
	keepalived	2.1.5
	local-volume-provisioner	2.5.0-mcp
	mariadb	10.4.17-bionic-20220113085105
IAM ^Updated	iam	2.4.31
	iam-controller	1.33.5
	keycloak	18.0.0
Container Cloud	admission-controller ^Updated	1.33.5
	agent-controller ^Updated	1.33.5
	byo-credentials-controller ^Updated	1.33.5
	byo-provider ^Updated	1.33.5
	ceph-kcc-controller ^Updated	1.33.5
	cert-manager ^Updated	1.33.5
	client-certificate-controller ^Updated	1.33.5
	golang	1.17.6
	event-controller ^Updated	1.33.5
	kaas-public-api ^Updated	1.33.5
	kaas-exporter ^Updated	1.33.5
	kaas-ui ^Updated	1.33.6
	lcm-controller ^Updated	0.3.0-285-g8498abe0
	license-controller ^Updated	1.33.5
	machinepool-controller ^Updated	1.33.5
	mcc-cache ^Updated	1.33.5
	portforward-controller ^Updated	1.33.5
	proxy-controller ^Updated	1.33.5
	rbac-controller ^Updated	1.33.5
	release-controller ^Updated	1.33.5
	rhellicense-controller ^Updated	1.33.5
	scope-controller ^Updated	1.33.5
	user-controller ^Updated	1.33.5
Equinix Metal	equinix-provider ^Updated	1.33.5
	equinix-credentials-controller ^Updated	1.33.5
	keepalived	2.1.5
OpenStack ^Updated	openstack-provider	1.33.5
	os-credentials-controller	1.33.5
VMware vSphere	vsphere-provider ^Updated	1.33.7
	vsphere-credentials-controller ^Updated	1.33.5
	keepalived	2.1.5
	squid-proxy	0.0.1-6

Artifacts¶

This section lists the components artifacts of the Mirantis Container Cloud release 2.20.0.

Bare metal artifacts
Core artifacts
IAM artifacts

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶
Artifact	Component	Path
Binaries	baremetal-operator ^Updated	https://binary.mirantis.com/bm/helm/baremetal-operator-6.3.3.tgz
	baremetal-public-api ^Updated	https://binary.mirantis.com/bm/helm/baremetal-public-api-6.3.3.tgz
	ironic-python-agent.initramfs ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-yoga-focal-debug-20220801150933
	ironic-python-agent.kernel ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-yoga-focal-debug-20220801150933
	kaas-ipam ^Updated	https://binary.mirantis.com/bm/helm/kaas-ipam-6.3.3.tgz
	local-volume-provisioner	https://binary.mirantis.com/bm/helm/local-volume-provisioner-2.5.0-mcp.tgz
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-104-6e2e82c.tgz
	target ubuntu system	https://binary.mirantis.com/bm/bin/efi/ubuntu/tgz-bionic-20210622161844
Docker images	ambassador	mirantis.azurecr.io/general/external/docker.io/library/nginx:1.20.1-alpine
	baremetal-operator	mirantis.azurecr.io/bm/baremetal-operator:base-focal-20220611131433
	baremetal-resource-controller ^Updated	mirantis.azurecr.io/bm/baremetal-resource-controller:base-focal-20220627134752
	dynamic_ipxe ^Updated	mirantis.azurecr.io/bm/dynamic-ipxe:base-focal-20220805114906
	dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-focal-20220705175454
	dnsmasq-controller ^Updated	mirantis.azurecr.io/bm/dnsmasq-controller:base-focal-20220704102028
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:yoga-focal-20220719132049
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:yoga-focal-20220719132049
	ironic-operator ^Removed	n/a
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20220602121226
	kaas-ipam	mirantis.azurecr.io/bm/kaas-ipam:base-focal-20220503165133
	mariadb	mirantis.azurecr.io/general/mariadb:10.4.17-bionic-20220113085105
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.14.0-1-g8725814
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-focal-20220128103433

Core artifacts¶
Artifact	Component	Paths
Bootstrap tarball ^Updated	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.33.5.tar.gz
	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.33.5.tar.gz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.33.5.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.33.5.tgz
	aws-credentials-controller	https://binary.mirantis.com/core/helm/aws-credentials-controller-1.33.5.tgz
	aws-provider	https://binary.mirantis.com/core/helm/aws-provider-1.33.5.tgz
	azure-credentials-controller	https://binary.mirantis.com/core/helm/azure-credentials-controller-1.33.5.tgz
	azure-provider	https://binary.mirantis.com/core/helm/azure-provider-1.33.5.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.33.5.tgz
	byo-credentials-controller	https://binary.mirantis.com/core/helm/byo-credentials-controller-1.33.5.tgz
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.33.5.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.33.5.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.33.5.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.33.5.tgz
	equinix-credentials-controller	https://binary.mirantis.com/core/helm/equinix-credentials-controller-1.33.5.tgz
	equinix-provider	https://binary.mirantis.com/core/helm/equinix-provider-1.33.5.tgz
	equinixmetalv2-provider	https://binary.mirantis.com/core/helm/equinixmetalv2-provider-1.33.5.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.33.5.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.33.5.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.33.5.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.33.5.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.33.6.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.33.5.tgz
	license-controller ^Updated	https://binary.mirantis.com/core/helm/license-controller-1.33.5.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.33.5.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.33.5.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.33.5.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.33.5.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.33.5.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.33.5.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.33.5.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.33.5.tgz
	scope-controller	http://binary.mirantis.com/core/helm/scope-controller-1.33.5.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.33.5.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.33.5.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.33.7.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.33.5.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.33.5
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.33.5
	aws-cluster-api-controller ^Updated	mirantis.azurecr.io/core/aws-cluster-api-controller:1.33.5
	aws-credentials-controller ^Updated	mirantis.azurecr.io/core/aws-credentials-controller:1.33.5
	azure-cluster-api-controller ^Updated	mirantis.azurecr.io/core/azure-cluster-api-controller:1.33.5
	azure-credentials-controller ^Updated	mirantis.azurecr.io/core/azure-credentials-controller:1.33.5
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.33.5
	byo-credentials-controller ^Updated	mirantis.azurecr.io/core/byo-credentials-controller:1.33.5
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.33.5
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.6.1
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.33.5
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.33.5
	cluster-api-provider-equinix ^Updated	mirantis.azurecr.io/core/equinix-cluster-api-controller:1.33.5
	equinix-credentials-controller ^Updated	mirantis.azurecr.io/core/equinix-credentials-controller:1.33.5
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.33.5
	haproxy	mirantis.azurecr.io/lcm/mcc-haproxy:v0.17.0-8-g6ca89d5
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.33.5
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.33.5
	lcm-controller ^Updated	mirantis.azurecr.io/lcm/lcm-controller:v0.3.0-285-g8498abe0
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.33.5
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.14.0-1-g8725814
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.33.5
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.33.5
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.33.5
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.33.5
	registry	mirantis.azurecr.io/lcm/registry:2.7.1
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.33.5
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.33.5
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.33.5
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-6
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.33.5
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.33.5
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.33.5

IAM artifacts¶
Artifact	Component	Path
Binaries	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
Helm charts ^Updated	iam	http://binary.mirantis.com/iam/helm/iam-2.4.31.tgz
	iam-proxy	http://binary.mirantis.com/iam/helm/iam-proxy-0.2.13.tgz
	keycloak_proxy	http://binary.mirantis.com/core/helm/keycloak_proxy-1.33.5.tgz
Docker images	kubernetes-entrypoint	mirantis.azurecr.io/iam/external/kubernetes-entrypoint:v0.3.1
	mariadb	mirantis.azurecr.io/general/mariadb:10.4.16-bionic-20201105025052
	keycloak ^Updated	mirantis.azurecr.io/iam/keycloak:0.5.10
	keycloak-gatekeeper	mirantis.azurecr.io/iam/keycloak-gatekeeper:7.1.3-2

See also

2.19.0¶

The Mirantis Container Cloud GA release 2.19.0:

Introduces support for the Cluster release 11.3.0 that is based on Mirantis Container Runtime 20.10.11 and Mirantis Kubernetes Engine 3.5.3 with Kubernetes 1.21.
Introduces support for the Cluster release 7.9.0 that is based on Mirantis Container Runtime 20.10.11 and Mirantis Kubernetes Engine 3.4.9 with Kubernetes 1.20.
Supports the Cluster release 8.8.0 that is based on the Cluster release 7.8.0 and represents Mirantis OpenStack for Kubernetes (MOSK) 22.3.
Does not support greenfield deployments on deprecated Cluster releases 11.2.0, 8.6.0, and 7.8.0. Use the latest Cluster releases of the series instead.

Caution

Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.19.0.

Enhancements¶

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.19.0. For the list of enhancements in the Cluster releases 11.3.0 and 7.9.0 that are introduced by the Container Cloud release 2.19.0, see the Cluster releases (managed).

General availability support for machines upgrade order
Web UI support for booting an OpenStack machine from a volume
Modification of network configuration on machines
New format of log entries on management clusters
Extended and basic versions of logs
Removal of Helm v2 support from Helm Controller
Distribution selector for bare metal machines in web UI

General availability support for machines upgrade order¶

Implemented full support for the upgrade sequence of machines that allows prioritized machines to be upgraded first. You can now set the upgrade index on an existing machine or machine pool using the Container Cloud web UI.

Consider the following upgrade index specifics:

The first machine to upgrade is always one of the control plane machines with the lowest upgradeIndex. Other control plane machines are upgraded one by one according to their upgrade indexes.
If the Cluster spec dedicatedControlPlane field is false, worker machines are upgraded only after the upgrade of all control plane machines finishes. Otherwise, they are upgraded after the first control plane machine, concurrently with other control plane machines.
If several machines have the same upgrade index, they have the same priority during upgrade.
If the value is not set, the machine is automatically assigned a value of the upgrade index.

Learn more

Operations Guide: Change upgrade order of a machine or machine pool

Web UI support for booting an OpenStack machine from a volume¶

TechPreview

Implemented the Boot From Volume option for the OpenStack machine creation wizard in the Container Cloud web UI. The feature allows booting OpenStack-based machines from a block storage volume.

The feature is beneficial for clouds that do not have enough space on hypervisors. After enabling this option, the Cinder storage is used instead of the Nova storage.

Modification of network configuration on machines¶

TechPreview

Enabled the ability to modify existing network configuration on running bare metal clusters with a mandatory approval of new settings by an Infrastructure Operator. This validation is required to prevent accidental cluster failures due to misconfiguration.

After you make necessary network configuration changes in the required L2 template, you now need to approve the changes by setting the spec.netconfigUpdateAllow:true flag in each affected IpamHost object.

Caution

For MKE clusters that are part of MOSK infrastructure, the feature support will become available in one of the following Container Cloud releases.

Learn more

Operations Guide: Modify network configuration on an existing machine since 2.19.0

New format of log entries on management clusters¶

Implemented a new format of log entries for cluster and machine logs of a management cluster. Each log entry now contains a request ID that identifies chronology of actions performed on a cluster or machine. The feature applies to all supported cloud providers.

The new format is <providerType>.<objectName>.req:<requestID>. For example, bm.machine.req:374, bm.cluster.req:172.

<providerType> - provider name, possible values: aws, azure, os, bm, vsphere, equinix.
<objectName> - name of an object being processed by provider, possible values: cluster, machine.
<requestID> - request ID number that increases when a provider receives a request from Kubernetes about creating, updating, deleting an object. The request ID allows combining all operations performed with an object within one request. For example, the result of a machine creation, update of its statuses, and so on.

Learn more

Extended and basic versions of logs¶

Implemented the --extended flag for collecting the extended version of logs that contains system and MKE logs, logs from LCM Ansible and LCM Agent along with cluster events and Kubernetes resources description and logs. You can use this flag to collect logs on any cluster type.

Learn more

Operations Guide: Collect cluster logs

Distribution selector for bare metal machines in web UI¶

Added the Distribution field to the bare metal machine creation wizard in the Container Cloud web UI. The default operating system in the distribution list is Ubuntu 20.04.

Caution

Do not use the outdated Ubuntu 18.04 distribution on greenfield deployments but only on existing clusters based on Ubuntu 18.04.

Learn more

Operations Guide: Create a bare metal machine using web UI

Removal of Helm v2 support from Helm Controller¶

After switching all remaining OpenStack Helm releases from v2 to v3, dropped support for Helm v2 in Helm Controller and removed the Tiller image for all related components.

Addressed issues¶

The following issues have been addressed in the Mirantis Container Cloud release 2.19.0 along with the Cluster releases 11.3.0 and 7.9.0:

[16379, 23865] Fixed the issue that caused an Equinix-based management or managed cluster update to fail with the FailedAttachVolume and FailedMount warnings.
[24286] Fixed the issue wherein creation of a new Equinix-based managed cluster failed due to failure to release a new vRouter ID.
[24722] Fixed the issue that caused Ceph clusters to be broken on Equinix-based managed clusters deployed on a Container Cloud instance with a non-default (different from region-one) region configured.
[24806] Fixed the issue wherein the dhcp-option=tag parameters were not applied to dnsmasq.conf during the bootstrap of a bare metal management cluster with a multi-rack topology.
[17778] Fixed the issue wherein the Container Cloud web UI displayed the new release version while update for some nodes was still in progress.
[24676] Fixed the issue wherein the deployment of an Equinix-based management cluster failed with the following error message:
```
Failed waiting for OIDC configuration readiness: timed out waiting for the
condition
```
[25050] For security reasons, disabled the deprecated TLS v1.0 and v1.1 for the mcc-cache and kaas-ui Container Cloud services.
[25256] Optimized the number of simultaneous connections to etcd to be open during configuration of Calico policies.
[24914] Fixed the issue wherein Helm Controller was getting stuck during readiness checks due to the timeout for helmclient being not set.

[24317] Fixed a number of security vulnerabilities in the Container Cloud Docker images:

Updated the following Docker images to fix CVE-2022-24407 and CVE-2022-0778:

admission-controller
agent-controller
aws-cluster-api-controller
aws-credentials-controller
azure-cluster-api-controller
azure-credentials-controller
bootstrap-controller
byo-cluster-api-controller

byo-credentials-controller
ceph-kcc-controller
cluster-api-provider-baremetal
equinix-cluster-api-controller
equinix-credentials-controller
event-controller
iam-controller
imc-sync

kaas-exporter
kproxy
license-controller
machinepool-controller
openstack-cluster-api-controller
os-credentials-controller
portforward-controller
proxy-controller

rbac-controller
release-controller
rhellicense-controller
scope-controller
storage-discovery
user-controller
vsphere-cluster-api-controller
vsphere-credentials-controller

Updated aws-ebs-csi-driver to fix the following Amazon Linux Security Advisories:
Updated keycloak to fix the following security vulnerabilities:
Updated busybox, iam/api, iam/helm, and nginx to fix CVE-2022-28391
Updated frontend to fix CVE-2022-27404
Updated kube-proxy to fix CVE-2022-1292

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.19.0 including the Cluster releases 11.3.0 and 7.9.0.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

MKE
Bare metal
StackLight
Container Cloud web UI

MKE¶

[20651] A cluster deployment or update fails with not ready compose deployments¶

'not ready: deployments: kube-system/compose got 0/0 replicas, kube-system/compose-api
 got 0/0 replicas'
 ready: false
 type: Kubernetes

Workaround:

Disable Docker Content Trust (DCT):
1. Access the MKE web UI as admin.
2. Navigate to Admin > Admin Settings.
3. In the left navigation pane, click Docker Content Trust and disable it.
Restart the affected deployments such as calico-kube-controllers, compose, compose-api, coredns, and so on:
```
kubectl -n kube-system delete deployment <deploymentName>
```
Once done, the cluster deployment or update resumes.
Re-enable DCT.

Bare metal¶

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

[20736] Region deletion failure after regional deployment failure¶

If a baremetal-based regional cluster deployment fails before pivoting is done, the corresponding region deletion fails.

Workaround:

cluster
machine
baremetalhost
baremetalhostprofile
l2template
subnet
ipamhost
ipaddr

kubectl delete <objectName> -l kaas.mirantis.com/region=<regionName>

Warning

Do not use the same region name again after the regional cluster deployment failure since some objects that reference the region name may still exist.

StackLight¶

[27732-1] OpenSearch PVC size custom settings are dismissed during deployment¶

Fixed in 11.6.0 and 12.7.0

Note

This issue does not block the OpenSearch cluster operations if the default retention time is set. The default setting is usually enough for the capacity size of this cluster.

The issue may affect the following Cluster releases:

11.2.0 - 11.5.0
7.8.0 - 7.11.0
8.8.0 - 8.10.0, 12.5.0 (MOSK clusters)
10.2.4 - 10.8.1 (attached MKE 3.4.x clusters)
13.0.2 - 13.5.1 (attached MKE 3.5.x clusters)

To verify that the cluster is affected:

Note

In the commands below, substitute parameters enclosed in angle brackets to match the affected cluster values.

kubectl --kubeconfig=<managementClusterKubeconfigPath> \
-n <affectedClusterProjectName> \
get cluster <affectedClusterName> \
-o=jsonpath='{.spec.providerSpec.value.helmReleases[*].values.elasticsearch.persistentVolumeClaimSize}' | xargs echo config size:


kubectl --kubeconfig=<affectedClusterKubeconfigPath> \
-n stacklight get pvc -l 'app=opensearch-master' \
-o=jsonpath="{.items[*].status.capacity.storage}" | xargs echo capacity sizes:

The cluster is not affected if the configuration size value matches or is less than any capacity size. For example:

config size: 30Gi
capacity sizes: 30Gi 30Gi 30Gi

config size: 50Gi
capacity sizes: 100Gi 100Gi 100Gi

The cluster is affected if the configuration size is larger than any capacity size. For example:
```
config size: 200Gi
capacity sizes: 100Gi 100Gi 100Gi
```

Workaround for a new cluster creation:

Select from the following options:
- For a management or regional cluster, during the bootstrap procedure, open cluster.yaml.template for editing.
- For a managed cluster, open the Cluster object for editing.
  
  Caution
  
  For a managed cluster, use the Container Cloud API instead of the web UI for cluster creation.

In the opened .yaml file, add logging.persistentVolumeClaimSize along with elasticsearch.persistentVolumeClaimSize. For example:

apiVersion: cluster.k8s.io/v1alpha1
spec:
...
  providerSpec:
    value:
    ...
      helmReleases:
      - name: stacklight
        values:
          elasticsearch:
            persistentVolumeClaimSize: 100Gi
          logging:
            enabled: true
            persistentVolumeClaimSize: 100Gi

Continue the cluster deployment. The system will use the custom value set in logging.persistentVolumeClaimSize.

Caution

If elasticsearch.persistentVolumeClaimSize is absent in the .yaml file, the Admission Controller blocks the configuration update.

Workaround for an existing cluster:

Caution

StackLight alerts list firing during cluster update

Cluster size and outage probability level	Alert name	Label name and component
Any cluster with high probability	`KubeStatefulSetOutage`	`statefulset=opensearch-master`
	`KubeDeploymentOutage`	`deployment=opensearch-dashboards` `deployment=metricbeat`
Large cluster with average probability	`KubePodsNotReady` ^{Removed in 17.0.0, 16.0.0, and 14.1.0}	`created_by_name="opensearch-master"` `created_by_name="opensearch-dashboards"` `created_by_name="metricbeat-*"`
	`OpenSearchClusterStatusWarning`	n/a
	`OpenSearchNumberOfPendingTasks`	n/a
	`OpenSearchNumberOfInitializingShards`	n/a
	`OpenSearchNumberOfUnassignedShards` ^{Removed in 2.27.0 (17.2.0 and 16.2.0)}	n/a
Any cluster with low probability	`KubeStatefulSetReplicasMismatch`	`statefulset=opensearch-master`
	`KubeDeploymentReplicasMismatch`	`deployment=opensearch-dashboards` `deployment=metricbeat`

StackLight in non-HA mode with a non-expandable StorageClass and no LVP for OpenSearch PVCs

Warning

If data loss is acceptable, proceed with the workaround below.

Note

To verify whether a StorageClass is expandable:

kubectl -n stacklight get pvc | grep opensearch-master | awk '{print $6}' | xargs -I{} kubectl get storageclass {} -o yaml | grep 'allowVolumeExpansion: true'

A positive system response is allowVolumeExpansion: true. A negative system response is blank or false.

Scale down the opensearch-master StatefulSet with dependent resources to 0 and disable the elasticsearch-curator CronJob:

kubectl -n stacklight scale --replicas 0 statefulset opensearch-master

kubectl -n stacklight scale --replicas 0 deployment opensearch-dashboards

kubectl -n stacklight scale --replicas 0 deployment metricbeat

kubectl -n stacklight patch cronjobs elasticsearch-curator -p '{"spec" : {"suspend" : true }}'

Recreate the opensearch-master StatefulSet with the updated disk size:

kubectl get statefulset opensearch-master -o yaml -n stacklight | sed 's/storage: 30Gi/storage: <<pvc_size>>/g' > opensearch-master.yaml

kubectl -n stacklight delete statefulset opensearch-master

kubectl create -f opensearch-master.yaml

Replace <pvcSize> with the elasticsearch.persistentVolumeClaimSize value.

Delete existing PVCs:
```
kubectl delete pvc -l 'app=opensearch-master' -n stacklight
```
Warning

This command removes all existing logs data from PVCs.

In the Cluster configuration, set logging.persistentVolumeClaimSize to the same value as the size of the elasticsearch.persistentVolumeClaimSize parameter. For example:

 apiVersion: cluster.k8s.io/v1alpha1
 kind: Cluster
 spec:
 ...
   providerSpec:
     value:
     ...
       helmReleases:
       - name: stacklight
         values:
           elasticsearch:
             persistentVolumeClaimSize: 100Gi
           logging:
             enabled: true
             persistentVolumeClaimSize: 100Gi

Scale up the opensearch-master StatefulSet with dependent resources to 1 and enable the elasticsearch-curator CronJob:

kubectl -n stacklight scale --replicas 1 statefulset opensearch-master

sleep 100

kubectl -n stacklight scale --replicas 1 deployment opensearch-dashboards

kubectl -n stacklight scale --replicas 1 deployment metricbeat

kubectl -n stacklight patch cronjobs elasticsearch-curator -p '{"spec" : {"suspend" : false }}'

[27732-2] Custom settings for ‘elasticsearch.logstashRetentionTime’ are dismissed¶

Fixed in 11.6.0 and 12.7.0

Custom settings for the deprecated elasticsearch.logstashRetentionTime parameter are overwritten by the default setting set to 1 day.

The issue may affect the following Cluster releases with enabled elasticsearch.logstashRetentionTime:

11.2.0 - 11.5.0
7.8.0 - 7.11.0
8.8.0 - 8.10.0, 12.5.0 (MOSK clusters)
10.2.4 - 10.8.1 (attached MKE 3.4.x clusters)
13.0.2 - 13.5.1 (attached MKE 3.5.x clusters)

As a workaround, in the Cluster object, replace elasticsearch.logstashRetentionTime with elasticsearch.retentionTime that was implemented to replace the deprecated parameter. For example:

apiVersion: cluster.k8s.io/v1alpha1
kind: Cluster
spec:
  ...
  providerSpec:
    value:
    ...
      helmReleases:
      - name: stacklight
        values:
          elasticsearch:
            retentionTime:
              logstash: 10
              events: 10
              notifications: 10
          logging:
            enabled: true

For the StackLight configuration procedure and parameters description, refer to Configure StackLight.

[20876] StackLight pods get stuck with the ‘NodeAffinity failed’ error¶

Note

Moving forward, the workaround for this issue will be moved from Release Notes to Operations Guide: Troubleshoot StackLight.

The issue does not affect the StackLight services, all required StackLight pods migrate successfully except extra pods that are created and stuck during pod migration.

As a workaround, remove the stuck pods:

kubectl --kubeconfig <managedClusterKubeconfig> -n stacklight delete pod <stuckPodName>

Container Cloud web UI¶

[26416] Failure to upload an MKE client bundle during cluster attachment¶

Fixed in 7.11.0, 11.5.0 and 12.5.0

During attachment of an existing MKE cluster using the Container Cloud web UI, uploading of an MKE client bundle fails with a false-positive message about a successful uploading.

Workaround:

Select from the following options:

Fill in the required fields for the MKE client bundle manually.
In the Attach Existing MKE Cluster window, use upload MKE client bundle twice to upload ucp.bundle-admin.zip and ucp-docker-bundle.zip located in the first archive.

[23002] Inability to set a custom value for a predefined node label¶

Fixed in 7.11.0, 11.5.0 and 12.5.0

During machine creation using the Container Cloud web UI, a custom value for a node label cannot be set.

As a workaround, manually add the value to spec.providerSpec.value.nodeLabels in machine.yaml.

Components versions¶

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.19.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

**Container Cloud release components versions**¶
Component	Application/Service	Version
AWS ^Updated	aws-provider	1.32.4
	aws-credentials-controller	1.32.4
Azure ^Updated	azure-provider	1.32.4
	azure-credentials-controller	1.32.4
Bare metal	ambassador	1.20.1-alpine
	baremetal-operator ^Updated	6.2.8
	baremetal-public-api ^Updated	6.2.8
	baremetal-provider ^Updated	1.32.4
	baremetal-resource-controller ^Updated	base-focal-20220530195224
	ironic ^Updated	xena-focal-20220603085546
	ironic-operator ^Updated	base-focal-20220605090941
	kaas-ipam ^Updated	base-focal-20220503165133
	keepalived	2.1.5
	local-volume-provisioner	2.5.0-mcp
	mariadb	10.4.17-bionic-20220113085105
IAM	iam ^Updated	2.4.29
	iam-controller ^Updated	1.32.4
	keycloak	16.1.1
Container Cloud	admission-controller ^Updated	1.32.10
	agent-controller ^Updated	1.32.4
	byo-credentials-controller ^Updated	1.32.4
	byo-provider ^Updated	1.32.4
	ceph-kcc-controller ^Updated	1.32.8
	cert-manager ^Updated	1.32.4
	client-certificate-controller ^Updated	1.32.4
	event-controller ^Updated	1.32.4
	golang	1.17.6
	kaas-public-api ^Updated	1.32.4
	kaas-exporter ^Updated	1.32.4
	kaas-ui ^Updated	1.32.10
	lcm-controller ^Updated	0.3.0-257-ga93244da
	license-controller ^Updated	1.32.4
	machinepool-controller ^Updated	1.32.4
	mcc-cache ^Updated	1.32.4
	portforward-controller ^Updated	1.32.4
	proxy-controller ^Updated	1.32.4
	rbac-controller ^Updated	1.32.4
	release-controller ^Updated	1.32.4
	rhellicense-controller ^Updated	1.32.4
	scope-controller ^Updated	1.32.4
	user-controller ^Updated	1.32.4
Equinix Metal	equinix-provider ^Updated	1.32.4
	equinix-credentials-controller ^Updated	1.32.4
	keepalived	2.1.5
OpenStack ^Updated	openstack-provider	1.32.4
	os-credentials-controller	1.32.4
VMware vSphere	vsphere-provider ^Updated	1.32.4
	vsphere-credentials-controller ^Updated	1.32.4
	keepalived	2.1.5
	squid-proxy	0.0.1-6

Artifacts¶

This section lists the components artifacts of the Mirantis Container Cloud release 2.19.0.

Bare metal artifacts
Core artifacts
IAM artifacts

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶
Artifact	Component	Path
Binaries	baremetal-operator ^Updated	https://binary.mirantis.com/bm/helm/baremetal-operator-6.2.8.tgz
	baremetal-public-api ^Updated	https://binary.mirantis.com/bm/helm/baremetal-public-api-6.2.8.tgz
	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-xena-focal-debug-20220512084815
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-xena-focal-debug-20220512084815
	kaas-ipam ^Updated	https://binary.mirantis.com/bm/helm/kaas-ipam-6.2.8.tgz
	local-volume-provisioner	https://binary.mirantis.com/bm/helm/local-volume-provisioner-2.5.0-mcp.tgz
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-104-6e2e82c.tgz
	target ubuntu system	https://binary.mirantis.com/bm/bin/efi/ubuntu/tgz-bionic-20210622161844
Docker images	ambassador	mirantis.azurecr.io/general/external/docker.io/library/nginx:1.20.1-alpine
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-focal-20220611131433
	baremetal-resource-controller ^Updated	mirantis.azurecr.io/bm/baremetal-resource-controller:base-focal-20220530195224
	dynamic_ipxe	mirantis.azurecr.io/bm/dynamic-ipxe:base-focal-20220429170829
	dnsmasq ^Updated	mirantis.azurecr.io/bm/baremetal-dnsmasq:base-focal-20220518104155
	dnsmasq-controller ^Updated	mirantis.azurecr.io/bm/dnsmasq-controller:base-focal-20220620190158
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:xena-focal-20220603085546
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:xena-focal-20220603085546
	ironic-operator ^Updated	mirantis.azurecr.io/bm/ironic-operator:base-focal-20220605090941
	ironic-prometheus-exporter ^Updated	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20220602121226
	kaas-ipam ^Updated	mirantis.azurecr.io/bm/kaas-ipam:base-focal-20220503165133
	mariadb	mirantis.azurecr.io/general/mariadb:10.4.17-bionic-20220113085105
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.14.0-1-g8725814
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-focal-20220128103433

Core artifacts¶
Artifact	Component	Paths
Bootstrap tarball ^Updated	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.32.4.tar.gz
	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.32.4.tar.gz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.32.10.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.32.4.tgz
	aws-credentials-controller	https://binary.mirantis.com/core/helm/aws-credentials-controller-1.32.4.tgz
	aws-provider	https://binary.mirantis.com/core/helm/aws-provider-1.32.4.tgz
	azure-credentials-controller	https://binary.mirantis.com/core/helm/azure-credentials-controller-1.32.4.tgz
	azure-provider	https://binary.mirantis.com/core/helm/azure-provider-1.32.4.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.32.4.tgz
	byo-credentials-controller	https://binary.mirantis.com/core/helm/byo-credentials-controller-1.32.4.tgz
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.32.4.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.32.4.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.32.4.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.32.4.tgz
	equinix-credentials-controller	https://binary.mirantis.com/core/helm/equinix-credentials-controller-1.32.4.tgz
	equinix-provider	https://binary.mirantis.com/core/helm/equinix-provider-1.32.4.tgz
	equinixmetalv2-provider	https://binary.mirantis.com/core/helm/equinixmetalv2-provider-1.32.4.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.32.4.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.32.4.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.32.4.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.32.4.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.32.10.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.32.4.tgz
	license-controller ^Updated	https://binary.mirantis.com/core/helm/license-controller-1.32.4.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.32.4.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.32.4.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.32.4.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.32.4.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.32.4.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.32.4.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.32.4.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.32.4.tgz
	scope-controller	http://binary.mirantis.com/core/helm/scope-controller-1.32.4.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.32.4.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.32.4.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.32.4.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.32.4.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.32.10
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.32.4
	aws-cluster-api-controller ^Updated	mirantis.azurecr.io/core/aws-cluster-api-controller:1.32.4
	aws-credentials-controller ^Updated	mirantis.azurecr.io/core/aws-credentials-controller:1.32.4
	azure-cluster-api-controller ^Updated	mirantis.azurecr.io/core/azure-cluster-api-controller:1.32.4
	azure-credentials-controller ^Updated	mirantis.azurecr.io/core/azure-credentials-controller:1.32.4
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.32.4
	byo-credentials-controller ^Updated	mirantis.azurecr.io/core/byo-credentials-controller:1.32.4
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:1.32.8
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.6.1
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.32.4
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.32.4
	cluster-api-provider-equinix ^Updated	mirantis.azurecr.io/core/equinix-cluster-api-controller:1.32.4
	equinix-credentials-controller ^Updated	mirantis.azurecr.io/core/equinix-credentials-controller:1.32.4
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.32.10
	haproxy ^Updated	mirantis.azurecr.io/lcm/mcc-haproxy:v0.17.0-8-g6ca89d5
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.32.4
	kproxy ^Updated	mirantis.azurecr.io/core/kproxy:1.32.4
	lcm-controller ^Updated	mirantis.azurecr.io/lcm/lcm-controller:v0.3.0-257-ga93244da
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.32.4
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.14.0-1-g8725814
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.32.4
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.32.4
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.32.4
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.32.4
	registry	mirantis.azurecr.io/lcm/registry:2.7.1
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.32.4
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.32.4
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.32.4
	squid-proxy	mirantis.azurecr.io/lcm/squid-proxy:0.0.1-6
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-cluster-api-controller:1.32.4
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.32.4
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.32.4

IAM artifacts¶
Artifact	Component	Path
Binaries	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
Helm charts	iam ^Updated	http://binary.mirantis.com/iam/helm/iam-2.4.29.tgz
	iam-proxy	http://binary.mirantis.com/iam/helm/iam-proxy-0.2.12.tgz
	keycloak_proxy ^Updated	http://binary.mirantis.com/core/helm/keycloak_proxy-1.32.4.tgz
Docker images	kubernetes-entrypoint	mirantis.azurecr.io/iam/external/kubernetes-entrypoint:v0.3.1
	mariadb	mirantis.azurecr.io/general/mariadb:10.4.16-bionic-20201105025052
	keycloak ^Updated	mirantis.azurecr.io/iam/keycloak:0.5.8
	keycloak-gatekeeper	mirantis.azurecr.io/iam/keycloak-gatekeeper:7.1.3-2

See also

2.18.1¶

The Mirantis Container Cloud GA release 2.18.1 is based on 2.18.0 and:

Introduces support for the Cluster release 8.8.0 that is based on the Cluster release 7.8.0 and represents Mirantis OpenStack for Kubernetes (MOSK) 22.3. This Cluster release is based on the updated version of Mirantis Kubernetes Engine 3.4.8 with Kubernetes 1.20 and Mirantis Container Runtime 20.10.11.
Supports the latest Cluster releases 7.8.0 and 11.2.0.
Does not support new deployments based on the deprecated Cluster releases 11.1.0, 8.6.0, and 7.7.0.

For details about the Container Cloud release 2.18.1, refer to its parent release 2.18.0:

Caution

See also

2.18.0¶

The Mirantis Container Cloud GA release 2.18.0:

Introduces support for the Cluster release 11.2.0 that is based on Mirantis Container Runtime 20.10.8 and Mirantis Kubernetes Engine 3.5.1 with Kubernetes 1.21.
Introduces support for the Cluster release 7.8.0 that is based on Mirantis Container Runtime 20.10.8 and Mirantis Kubernetes Engine 3.4.7 with Kubernetes 1.20.
Supports the Cluster release 8.6.0 that is based on the Cluster release 7.6.0 and represents Mirantis OpenStack for Kubernetes (MOSK) 22.2.
Does not support greenfield deployments on deprecated Cluster releases 11.1.0, 8.5.0, and 7.7.0. Use the latest Cluster releases of the series instead.

Caution

Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.18.0.

Enhancements¶

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.18.0. For the list of enhancements in the Cluster releases 11.2.0 and 7.8.0 that are introduced by the Container Cloud release 2.18.0, see the Cluster releases (managed).

Ubuntu kernel update for bare metal clusters
Support for Ubuntu 20.04 on greenfield vSphere deployments
Booting a machine from a block storage volume for OpenStack provider
IPSec encryption for the Kubernetes workloads network
Support for MITM proxy
Support for custom Docker registries
Upgrade sequence for machines
Enablement of Salesforce propagation to all clusters using web UI
Documentation enhancements

Ubuntu kernel update for bare metal clusters¶

Updated the Ubuntu kernel version to 5.4.0-109-generic for bare metal non-MOSK-based management, regional, and managed clusters to apply Ubuntu 18.04 or 20.04 security and system updates.

Caution

During a baremetal-based cluster update to Container Cloud 2.18 and to the latest Cluster releases 11.2.0 and 7.8.0, hosts will be restarted to apply the latest supported Ubuntu 18.04 or 20.04 packages. Therefore:

Depending on the cluster configuration, applying security updates and host restart can increase the update time for each node to up to 1 hour.
Cluster nodes are updated one by one. Therefore, for large clusters, the update may take several days to complete.

Support for Ubuntu 20.04 on greenfield vSphere deployments¶

Implemented full support for Ubuntu 20.04 LTS (Focal Fossa) as the default host operating system that now installs on management, regional, and managed clusters for the vSphere cloud provider.

Caution

Upgrading from Ubuntu 18.04 to 20.04 on existing deployments is not supported.

Booting a machine from a block storage volume for OpenStack provider¶

TechPreview

Implemented initial Technology Preview support for booting of the OpenStack-based machines from a block storage volume. The feature is beneficial for clouds that do not have enough space on hypervisors. After enabling this option, the Cinder storage is used instead of the Nova storage.

Using the Container Cloud API, you can boot the Bastion node, or the required management, regional, or managed cluster nodes from a volume.

Note

The ability to enable the boot from volume option using the Container Cloud web UI for managed clusters will be implemented in one of the following Container Cloud releases.

IPSec encryption for the Kubernetes workloads network¶

TechPreview Experimental since 2.19.0

Implemented initial Technology Preview support for enabling IPSec encryption for the Kubernetes workloads network. The feature allows for secure communication between servers.

You can enable encryption for the Kubernetes workloads network on greenfield deployments during initial creation of a management, regional, and managed cluster through the Cluster object using the secureOverlay parameter.

Caution

For the Azure cloud provider, the feature is not supported. For details, see MKE documentation: Kubernetes network encryption.
For the bare metal cloud provider and MOSK-based deployments, the feature support will become available in one of the following Container Cloud releases.
For existing deployments, the feature support will become available in one of the following Container Cloud releases.

Learn more

Support for MITM proxy¶

TechPreview

Implemented the initial Technology Preview support for man-in-the-middle (MITM) proxies on offline OpenStack and non-MOSK-based bare metal deployments. Using trusted proxy CA certificates, the feature allows monitoring all cluster traffic for security and audit purposes.

Learn more

Support for custom Docker registries¶

Implemented support for custom Docker registries configuration in the Container Cloud management, regional, and managed clusters. Using the ContainerRegistry custom resource, you can configure CA certificates on machines to access private Docker registries.

Note

For MOSK-based deployments, the feature support is available since Container Cloud 2.18.1.

Learn more

Upgrade sequence for machines¶

TechPreview

Implemented initial Technology Preview support for machines upgrade index that allows prioritized machines to be upgraded first. During a machine or a machine pool creation, you can use the Container Cloud web UI Upgrade Index option to set a positive numeral value that defines the order of machine upgrade during cluster update.

To set the upgrade order on an existing cluster, use the Container Cloud API:

For a machine that is not assigned to a machine pool, add the upgradeIndex field with the required value to the spec:providerSpec:value section in the Machine object.
For a machine pool, add the upgradeIndex field with the required value to the spec:machineSpec:providerSpec:value section of the MachinePool object to apply the upgrade order to all machines in the pool.

Note

The first machine to upgrade is always one of the control plane machines with the lowest upgradeIndex. Other control plane machines are upgraded one by one according to their upgrade indexes. If the Cluster spec dedicatedControlPlane field is false, worker machines are upgraded only after the upgrade of all control plane machines finishes. Otherwise, they are upgraded after the first control plane machine, concurrently with other control plane machines.
If two or more machines have the same value of upgradeIndex, these machines are equally prioritized during upgrade.
Changing of the machine upgrade index during an already running cluster update or maintenance is not supported.

Learn more

Operations Guide: Add a machine for selected cloud provider

Enablement of Salesforce propagation to all clusters using web UI¶

Simplified the ability to enable automatic update and sync of the Salesforce configuration on all your clusters by adding the corresponding check box to the Salesforce settings in the Container Cloud web UI.

Learn more

Operations Guide: Automatically propagate Salesforce configuration to all clusters

Documentation enhancements¶

On top of continuous improvements delivered to the existing Container Cloud guides, added the following documentation:

Addressed issues¶

The following issues have been addressed in the Mirantis Container Cloud release 2.18.0 along with the Cluster releases 11.2.0 and 7.8.0:

[24075] Fixed the issue with the Ubuntu 20.04 option not displaying in the operating systems drop-down list during machine creation for the AWS and Equinix Metal with public networking providers.

Warning

After Container Cloud is upgraded to 2.18.0, remove the values added during the workaround application from the Cluster object.
[9339] Fixed the issue with incorrect health monitoring for Kubernetes and MKE endpoints on OpenStack-based clusters.
[21710] Fixed the issue with a too high threshold being set for the KubeContainersCPUThrottlingHigh StackLight alert.
[22872] Removed the inefficient ElasticNoNewDataCluster and ElasticNoNewDataNode StackLight alerts.
[23853] Fixed the issue wherein the KaaSCephOperationRequest resource created to remove the failed node from the Ceph cluster was stuck with the Failed status and an error message in errorReason. The Failed status blocked the replacement of the failed master node on regional clusters of the bare metal and Equinix Metal providers.
[23841] Improved error logging for load balancers deletion:
- The reason for the inability to delete an LB is now displayed in the provider logs.
- If the search of an FIP associated with the LB deletion returns more than one FIP, the provider returns an error instead of deleting all found FIPs.
[18331] Fixed the issue with the Keycloak admin console menu disappearing on the Add identity provider page during configuration of an identity provider SAML.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.18.0 including the Cluster releases 11.2.0 and 7.8.0.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

MKE
Bare metal
Equinix Metal
Upgrade
StackLight
Container Cloud web UI

MKE¶

[20651] A cluster deployment or update fails with not ready compose deployments¶

'not ready: deployments: kube-system/compose got 0/0 replicas, kube-system/compose-api
 got 0/0 replicas'
 ready: false
 type: Kubernetes

Workaround:

Disable Docker Content Trust (DCT):
1. Access the MKE web UI as admin.
2. Navigate to Admin > Admin Settings.
3. In the left navigation pane, click Docker Content Trust and disable it.
Restart the affected deployments such as calico-kube-controllers, compose, compose-api, coredns, and so on:
```
kubectl -n kube-system delete deployment <deploymentName>
```
Once done, the cluster deployment or update resumes.
Re-enable DCT.

Bare metal¶

[24806] The dnsmasq parameters are not applied on multi-rack clusters¶

Fixed in 2.19.0

During bootstrap of a bare metal management cluster with a multi-rack topology, the dhcp-option=tag parameters are not applied to dnsmasq.conf.

Symptoms:

The dnasmq-controller service contains the following exemplary error message:

KUBECONFIG=kaas-mgmt-kubeconfig kubectl -n kaas logs --tail 50 deployment/dnsmasq -c dnsmasq-controller

...
I0622 09:05:26.898898       8 handler.go:19] Failed to watch Object, kind:'dnsmasq': failed to list *unstructured.Unstructured: the server could not find the requested resource
E0622 09:05:26.899108       8 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.8/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
...

Workaround:

Manually update deployment/dnsmasq with the updated image:

KUBECONFIG=kaas-mgmt-kubeconfig kubectl -n kaas set image deployment/dnsmasq dnsmasq-controller=mirantis.azurecr.io/bm/dnsmasq-controller:base-focal-2-18-issue24806-20220618085127

[24005] Deletion of a node with ironic Pod is stuck in the Terminating state¶

During deletion of a manager machine running the ironic Pod from a bare metal management cluster, the following problems occur:

All Pods are stuck in the Terminating state
A new ironic Pod fails to start
The related bare metal host is stuck in the deprovisioning state

As a workaround, before deletion of the node running the ironic Pod, cordon and drain the node using the kubectl cordon <nodeName> and kubectl drain <nodeName> commands.

[20736] Region deletion failure after regional deployment failure¶

If a baremetal-based regional cluster deployment fails before pivoting is done, the corresponding region deletion fails.

Workaround:

cluster
machine
baremetalhost
baremetalhostprofile
l2template
subnet
ipamhost
ipaddr

kubectl delete <objectName> -l kaas.mirantis.com/region=<regionName>

Warning

Do not use the same region name again after the regional cluster deployment failure since some objects that reference the region name may still exist.

Equinix Metal¶

[16379,23865] Cluster update fails with the FailedMount warning¶

Fixed in 2.19.0

An Equinix-based management or managed cluster fails to update with the FailedAttachVolume and FailedMount warnings.

Workaround:

Verify that the description of the pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
- <affectedProjectName> is the Container Cloud project name where the pods failed to run
- <affectedPodName> is a pod name that failed to run in this project
In the pod description, identify the node name where the pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the pod that fails to init to 0 replicas.

On every csi-rbdplugin pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state is Running.

StackLight¶

[27732-1] OpenSearch PVC size custom settings are dismissed during deployment¶

Fixed in 11.6.0 and 12.7.0

Note

This issue does not block the OpenSearch cluster operations if the default retention time is set. The default setting is usually enough for the capacity size of this cluster.

The issue may affect the following Cluster releases:

11.2.0 - 11.5.0
7.8.0 - 7.11.0
8.8.0 - 8.10.0, 12.5.0 (MOSK clusters)
10.2.4 - 10.8.1 (attached MKE 3.4.x clusters)
13.0.2 - 13.5.1 (attached MKE 3.5.x clusters)

To verify that the cluster is affected:

Note

In the commands below, substitute parameters enclosed in angle brackets to match the affected cluster values.

kubectl --kubeconfig=<managementClusterKubeconfigPath> \
-n <affectedClusterProjectName> \
get cluster <affectedClusterName> \
-o=jsonpath='{.spec.providerSpec.value.helmReleases[*].values.elasticsearch.persistentVolumeClaimSize}' | xargs echo config size:


kubectl --kubeconfig=<affectedClusterKubeconfigPath> \
-n stacklight get pvc -l 'app=opensearch-master' \
-o=jsonpath="{.items[*].status.capacity.storage}" | xargs echo capacity sizes:

The cluster is not affected if the configuration size value matches or is less than any capacity size. For example:

config size: 30Gi
capacity sizes: 30Gi 30Gi 30Gi

config size: 50Gi
capacity sizes: 100Gi 100Gi 100Gi

The cluster is affected if the configuration size is larger than any capacity size. For example:
```
config size: 200Gi
capacity sizes: 100Gi 100Gi 100Gi
```

Workaround for a new cluster creation:

Select from the following options:
- For a management or regional cluster, during the bootstrap procedure, open cluster.yaml.template for editing.
- For a managed cluster, open the Cluster object for editing.
  
  Caution
  
  For a managed cluster, use the Container Cloud API instead of the web UI for cluster creation.

In the opened .yaml file, add logging.persistentVolumeClaimSize along with elasticsearch.persistentVolumeClaimSize. For example:

apiVersion: cluster.k8s.io/v1alpha1
spec:
...
  providerSpec:
    value:
    ...
      helmReleases:
      - name: stacklight
        values:
          elasticsearch:
            persistentVolumeClaimSize: 100Gi
          logging:
            enabled: true
            persistentVolumeClaimSize: 100Gi

Continue the cluster deployment. The system will use the custom value set in logging.persistentVolumeClaimSize.

Caution

If elasticsearch.persistentVolumeClaimSize is absent in the .yaml file, the Admission Controller blocks the configuration update.

Workaround for an existing cluster:

Caution

StackLight alerts list firing during cluster update

Cluster size and outage probability level	Alert name	Label name and component
Any cluster with high probability	`KubeStatefulSetOutage`	`statefulset=opensearch-master`
	`KubeDeploymentOutage`	`deployment=opensearch-dashboards` `deployment=metricbeat`
Large cluster with average probability	`KubePodsNotReady` ^{Removed in 17.0.0, 16.0.0, and 14.1.0}	`created_by_name="opensearch-master"` `created_by_name="opensearch-dashboards"` `created_by_name="metricbeat-*"`
	`OpenSearchClusterStatusWarning`	n/a
	`OpenSearchNumberOfPendingTasks`	n/a
	`OpenSearchNumberOfInitializingShards`	n/a
	`OpenSearchNumberOfUnassignedShards` ^{Removed in 2.27.0 (17.2.0 and 16.2.0)}	n/a
Any cluster with low probability	`KubeStatefulSetReplicasMismatch`	`statefulset=opensearch-master`
	`KubeDeploymentReplicasMismatch`	`deployment=opensearch-dashboards` `deployment=metricbeat`

StackLight in non-HA mode with a non-expandable StorageClass and no LVP for OpenSearch PVCs

Warning

If data loss is acceptable, proceed with the workaround below.

Note

To verify whether a StorageClass is expandable:

kubectl -n stacklight get pvc | grep opensearch-master | awk '{print $6}' | xargs -I{} kubectl get storageclass {} -o yaml | grep 'allowVolumeExpansion: true'

A positive system response is allowVolumeExpansion: true. A negative system response is blank or false.

Scale down the opensearch-master StatefulSet with dependent resources to 0 and disable the elasticsearch-curator CronJob:

kubectl -n stacklight scale --replicas 0 statefulset opensearch-master

kubectl -n stacklight scale --replicas 0 deployment opensearch-dashboards

kubectl -n stacklight scale --replicas 0 deployment metricbeat

kubectl -n stacklight patch cronjobs elasticsearch-curator -p '{"spec" : {"suspend" : true }}'

Recreate the opensearch-master StatefulSet with the updated disk size:

kubectl get statefulset opensearch-master -o yaml -n stacklight | sed 's/storage: 30Gi/storage: <<pvc_size>>/g' > opensearch-master.yaml

kubectl -n stacklight delete statefulset opensearch-master

kubectl create -f opensearch-master.yaml

Replace <pvcSize> with the elasticsearch.persistentVolumeClaimSize value.

Delete existing PVCs:
```
kubectl delete pvc -l 'app=opensearch-master' -n stacklight
```
Warning

This command removes all existing logs data from PVCs.

In the Cluster configuration, set logging.persistentVolumeClaimSize to the same value as the size of the elasticsearch.persistentVolumeClaimSize parameter. For example:

 apiVersion: cluster.k8s.io/v1alpha1
 kind: Cluster
 spec:
 ...
   providerSpec:
     value:
     ...
       helmReleases:
       - name: stacklight
         values:
           elasticsearch:
             persistentVolumeClaimSize: 100Gi
           logging:
             enabled: true
             persistentVolumeClaimSize: 100Gi

Scale up the opensearch-master StatefulSet with dependent resources to 1 and enable the elasticsearch-curator CronJob:

kubectl -n stacklight scale --replicas 1 statefulset opensearch-master

sleep 100

kubectl -n stacklight scale --replicas 1 deployment opensearch-dashboards

kubectl -n stacklight scale --replicas 1 deployment metricbeat

kubectl -n stacklight patch cronjobs elasticsearch-curator -p '{"spec" : {"suspend" : false }}'

[27732-2] Custom settings for ‘elasticsearch.logstashRetentionTime’ are dismissed¶

Fixed in 11.6.0 and 12.7.0

Custom settings for the deprecated elasticsearch.logstashRetentionTime parameter are overwritten by the default setting set to 1 day.

The issue may affect the following Cluster releases with enabled elasticsearch.logstashRetentionTime:

11.2.0 - 11.5.0
7.8.0 - 7.11.0
8.8.0 - 8.10.0, 12.5.0 (MOSK clusters)
10.2.4 - 10.8.1 (attached MKE 3.4.x clusters)
13.0.2 - 13.5.1 (attached MKE 3.5.x clusters)

As a workaround, in the Cluster object, replace elasticsearch.logstashRetentionTime with elasticsearch.retentionTime that was implemented to replace the deprecated parameter. For example:

apiVersion: cluster.k8s.io/v1alpha1
kind: Cluster
spec:
  ...
  providerSpec:
    value:
    ...
      helmReleases:
      - name: stacklight
        values:
          elasticsearch:
            retentionTime:
              logstash: 10
              events: 10
              notifications: 10
          logging:
            enabled: true

For the StackLight configuration procedure and parameters description, refer to Configure StackLight.

[20876] StackLight pods get stuck with the ‘NodeAffinity failed’ error¶

Note

Moving forward, the workaround for this issue will be moved from Release Notes to Operations Guide: Troubleshoot StackLight.

The issue does not affect the StackLight services, all required StackLight pods migrate successfully except extra pods that are created and stuck during pod migration.

As a workaround, remove the stuck pods:

kubectl --kubeconfig <managedClusterKubeconfig> -n stacklight delete pod <stuckPodName>

Upgrade¶

[24802] Container Cloud upgrade to 2.18.0 can trigger managed clusters update¶

Affects only Container Cloud 2.18.0

On clusters with enabled proxy and the NO_PROXY settings containing localhost/127.0.0.1 or matching the automatically added Container Cloud internal endpoints, the Container Cloud release upgrade from 2.17.0 to 2.18.0 triggers automatic update of managed clusters to the latest available Cluster releases in their respective series.

For the issue workaround, contact Mirantis support.

[21810] Upgrade to Cluster releases 5.22.0 and 7.5.0 may get stuck¶

Affects Ubuntu-based clusters deployed after Feb 10, 2022

If you deploy an Ubuntu-based cluster using the deprecated Cluster release 7.4.0 (and earlier) or 5.21.0 (and earlier) starting from February 10, 2022, the cluster update to the Cluster releases 7.5.0 and 5.22.0 may get stuck while applying the Deploy state to the cluster machines. The issue affects all cluster types: management, regional, and managed.

To verify that the cluster is affected:

Log in to the Container Cloud web UI.
In the Clusters tab, capture the RELEASE and AGE values of the required Ubuntu-based cluster. If the values match the ones from the issue description, the cluster may be affected.
Using SSH, log in to the manager or worker node that got stuck while applying the Deploy state and identify the containerd package version:
```
containerd --version
```
If the version is 1.5.9, the cluster is affected.

In /var/log/lcm/runners/<nodeName>/deploy/, verify whether the Ansible deployment logs contain the following errors that indicate that the cluster is affected:

The following packages will be upgraded:
  docker-ee docker-ee-cli
The following packages will be DOWNGRADED:
  containerd.io

STDERR:
E: Packages were downgraded and -y was used without --allow-downgrades.

Workaround:

Warning

Apply the steps below to the affected nodes one-by-one and only after each consecutive node gets stuck on the Deploy phase with the Ansible log errors. Such sequence ensures that each node is cordon-drained and Docker is properly stopped. Therefore, no workloads are affected.

Using SSH, log in to the first affected node and install containerd 1.5.8:

apt-get install containerd.io=1.5.8-1 -y --allow-downgrades --allow-change-held-packages

Wait for Ansible to reconcile. The node should become Ready in several minutes.
Wait for the next node of the cluster to get stuck on the Deploy phase with the Ansible log errors. Only after that, apply the steps above on the next node.
Patch the remaining nodes one-by-one using the steps above.

Container Cloud web UI¶

[23002] Inability to set a custom value for a predefined node label¶

Fixed in 7.11.0, 11.5.0 and 12.5.0

During machine creation using the Container Cloud web UI, a custom value for a node label cannot be set.

As a workaround, manually add the value to spec.providerSpec.value.nodeLabels in machine.yaml.

[249] A newly created project does not display in the Container Cloud web UI¶

Affects only Container Cloud 2.18.0 and earlier

A project that is newly created in the Container Cloud web UI does not display in the Projects list even after refreshing the page. The issue occurs due to the token missing the necessary role for the new project. As a workaround, relogin to the Container Cloud web UI.

Components versions¶

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.18.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

**Container Cloud release components versions**¶
Component	Application/Service	Version
AWS ^Updated	aws-provider	1.31.9
	aws-credentials-controller	1.31.9
Azure ^Updated	azure-provider	1.31.9
	azure-credentials-controller	1.31.9
Bare metal	ambassador	1.20.1-alpine
	baremetal-operator ^Updated	6.1.9
	baremetal-public-api ^Updated	6.1.9
	baremetal-provider ^Updated	1.31.9
	baremetal-resource-controller	base-focal-20220429170738
	ironic ^Updated	xena-focal-20220513073431
	ironic-operator ^Updated	base-focal-20220501190529
	kaas-ipam	base-focal-20220310095439
	keepalived	2.1.5
	local-volume-provisioner	2.5.0-mcp
	mariadb	10.4.17-bionic-20220113085105
IAM	iam ^Updated	2.4.25
	iam-controller ^Updated	1.31.9
	keycloak ^Updated	16.1.1
Container Cloud	admission-controller ^Updated	1.31.11
	agent-controller ^Updated	1.31.9
	byo-credentials-controller ^Updated	1.31.9
	byo-provider ^Updated	1.31.9
	ceph-kcc-controller ^Updated	1.31.9
	cert-manager ^Updated	1.31.9
	client-certificate-controller ^Updated	1.31.9
	event-controller ^Updated	1.31.9
	golang	1.17.6
	kaas-public-api ^Updated	1.31.9
	kaas-exporter ^Updated	1.31.9
	kaas-ui ^Updated	1.31.12
	lcm-controller ^Updated	0.3.0-239-gae7218ea
	license-controller ^Updated	1.31.9
	machinepool-controller ^Updated	1.31.9
	mcc-cache ^Updated	1.31.9
	portforward-controller ^Updated	1.31.9
	proxy-controller ^Updated	1.31.9
	rbac-controller ^Updated	1.31.9
	release-controller ^Updated	1.31.9
	rhellicense-controller ^Updated	1.31.9
	scope-controller ^Updated	1.31.9
	squid-proxy	0.0.1-6
	user-controller ^Updated	1.31.9
Equinix Metal	equinix-provider ^Updated	1.31.9
	equinix-credentials-controller ^Updated	1.31.9
	keepalived	2.1.5
OpenStack ^Updated	openstack-provider	1.31.9
	os-credentials-controller	1.31.9
VMware vSphere	vsphere-provider ^Updated	1.31.9
	vsphere-credentials-controller ^Updated	1.31.9
	keepalived	2.1.5

Artifacts¶

This section lists the components artifacts of the Mirantis Container Cloud release 2.18.0.

Bare metal artifacts
Core artifacts
IAM artifacts

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶
Artifact	Component	Path
Binaries	baremetal-operator ^Updated	https://binary.mirantis.com/bm/helm/baremetal-operator-6.1.9.tgz
	baremetal-public-api ^Updated	https://binary.mirantis.com/bm/helm/baremetal-public-api-6.1.9.tgz
	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-xena-focal-debug-20220512084815
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-xena-focal-debug-20220512084815
	kaas-ipam ^Updated	https://binary.mirantis.com/bm/helm/kaas-ipam-6.1.9.tgz
	local-volume-provisioner	https://binary.mirantis.com/bm/helm/local-volume-provisioner-2.5.0-mcp.tgz
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-104-6e2e82c.tgz
	target ubuntu system	https://binary.mirantis.com/bm/bin/efi/ubuntu/tgz-bionic-20210622161844
Docker images	ambassador	mirantis.azurecr.io/general/external/docker.io/library/nginx:1.20.1-alpine
	baremetal-operator	mirantis.azurecr.io/bm/baremetal-operator:base-focal-20220208045851
	baremetal-resource-controller ^Updated	mirantis.azurecr.io/bm/baremetal-resource-controller:base-focal-20220429170738
	dynamic_ipxe ^Updated	mirantis.azurecr.io/bm/dnsmasq/dynamic-ipxe:base-focal-20220429170829
	dnsmasq ^Updated	mirantis.azurecr.io/general/dnsmasq:focal-20220429170747
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:xena-focal-20220513073431
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:xena-focal-20220513073431
	ironic-operator ^Updated	mirantis.azurecr.io/bm/ironic-operator:base-focal-20220501190529
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20210608113804
	kaas-ipam	mirantis.azurecr.io/bm/kaas-ipam:base-focal-20220310095439
	mariadb	mirantis.azurecr.io/general/mariadb:10.4.17-bionic-20220113085105
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.14.0-1-g8725814
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-focal-20220128103433

Core artifacts¶
Artifact	Component	Paths
Bootstrap tarball ^Updated	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.31.9.tar.gz
	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.31.9.tar.gz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.31.11.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.31.9.tgz
	aws-credentials-controller	https://binary.mirantis.com/core/helm/aws-credentials-controller-1.31.9.tgz
	aws-provider	https://binary.mirantis.com/core/helm/aws-provider-1.31.9.tgz
	azure-credentials-controller	https://binary.mirantis.com/core/helm/azure-credentials-controller-1.31.9.tgz
	azure-provider	https://binary.mirantis.com/core/helm/azure-provider-1.31.9.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.31.9.tgz
	byo-credentials-controller	https://binary.mirantis.com/core/helm/byo-credentials-controller-1.31.9.tgz
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.31.9.tgz
	ceph-kcc-controller	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.31.9.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.31.9.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.31.9.tgz
	equinix-credentials-controller	https://binary.mirantis.com/core/helm/equinix-credentials-controller-1.31.9.tgz
	equinix-provider	https://binary.mirantis.com/core/helm/equinix-provider-1.31.9.tgz
	equinixmetalv2-provider	https://binary.mirantis.com/core/helm/equinixmetalv2-provider-1.31.9.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.31.9.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.31.9.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.31.9.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.31.9.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.31.12.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.31.9.tgz
	license-controller ^Updated	https://binary.mirantis.com/core/helm/license-controller-1.31.9.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.31.9.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.31.9.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.31.9.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.31.9.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.31.9.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.31.9.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.31.9.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.31.9.tgz
	scope-controller	http://binary.mirantis.com/core/helm/scope-controller-1.31.9.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.31.9.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.31.9.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.31.9.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.31.9.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.31.11
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.31.9
	aws-cluster-api-controller ^Updated	mirantis.azurecr.io/core/aws-cluster-api-controller:1.31.9
	aws-credentials-controller ^Updated	mirantis.azurecr.io/core/aws-credentials-controller:1.31.9
	azure-cluster-api-controller ^Updated	mirantis.azurecr.io/core/azure-cluster-api-controller:1.31.9
	azure-credentials-controller ^Updated	mirantis.azurecr.io/core/azure-credentials-controller:1.31.9
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.31.9
	byo-credentials-controller ^Updated	mirantis.azurecr.io/core/byo-credentials-controller:1.31.9
	ceph-kcc-controller ^Updated	mirantis.azurecr.io/core/ceph-kcc-controller:v1.31.9
	cert-manager-controller	mirantis.azurecr.io/core/external/cert-manager-controller:v1.6.1
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.31.9
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.31.9
	cluster-api-provider-equinix ^Updated	mirantis.azurecr.io/core/cluster-api-provider-equinix:1.31.9
	equinix-credentials-controller ^Updated	mirantis.azurecr.io/core/equinix-credentials-controller:1.31.9
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.31.12
	haproxy	mirantis.azurecr.io/lcm/mcc-haproxy:v0.12.0-8-g6fabf1c
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.31.9
	kproxy ^Updated	mirantis.azurecr.io/lcm/kproxy:1.31.9
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:v0.3.0-239-gae7218ea
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.31.9
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.14.0-1-g8725814
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.31.9
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.31.9
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.31.9
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.31.9
	registry	mirantis.azurecr.io/lcm/registry:2.7.1
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.31.9
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.31.9
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.31.9
	squid-proxy	mirantis.azurecr.io/core/squid-proxy:0.0.1-6
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-api-controller:1.31.9
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.31.9
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.31.9

IAM artifacts¶
Artifact	Component	Path
Binaries	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	iamctl-linux ^Removed	n/a
	iamctl-darwin ^Removed	n/a
	iamctl-windows ^Removed	n/a
Helm charts	iam ^Updated	http://binary.mirantis.com/iam/helm/iam-2.4.25.tgz
	iam-proxy	http://binary.mirantis.com/iam/helm/iam-proxy-0.2.12.tgz
	keycloak_proxy ^Updated	http://binary.mirantis.com/core/helm/keycloak_proxy-1.31.9.tgz
Docker images	api ^Removed	n/a
	auxiliary ^Removed	n/a
	kubernetes-entrypoint	mirantis.azurecr.io/iam/external/kubernetes-entrypoint:v0.3.1
	mariadb	mirantis.azurecr.io/general/mariadb:10.4.16-bionic-20201105025052
	keycloak ^Updated	mirantis.azurecr.io/iam/keycloak:0.5.7
	keycloak-gatekeeper	mirantis.azurecr.io/iam/keycloak-gatekeeper:7.1.3-2

See also

2.17.0¶

The Mirantis Container Cloud GA release 2.17.0:

Introduces support for the Cluster release 11.1.0 that is based on Mirantis Container Runtime 20.10.8 and Mirantis Kubernetes Engine 3.5.1 with Kubernetes 1.21.
Introduces support for the Cluster release 7.7.0 that is based on Mirantis Container Runtime 20.10.8 and Mirantis Kubernetes Engine 3.4.7 with Kubernetes 1.20.
Supports the Cluster release 8.6.0 that is based on the Cluster release 7.6.0 and represents Mirantis OpenStack for Kubernetes (MOSK) 22.2.
Does not support greenfield deployments on deprecated Cluster releases 11.0.0, 8.5.0, and 7.6.0. Use the latest Cluster releases of the series instead.

Caution

Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.17.0.

Enhancements¶

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.17.0. For the list of enhancements in the Cluster releases 11.1.0 and 7.7.0 that are introduced by the Container Cloud release 2.17.0, see the Cluster releases (managed).

General availability for Ubuntu 20.04 on greenfield deployments
Container Cloud on top of MOSK Victoria with Tungsten Fabric
EBS instead of NVMe as persistent storage for AWS-based nodes
Manager nodes deletion on all cluster types
Custom values for node labels
Machine pools
Automatic propagation of Salesforce configuration to all clusters

General availability for Ubuntu 20.04 on greenfield deployments¶

Implemented full support for Ubuntu 20.04 LTS (Focal Fossa) as the default host operating system that now installs on management, regional, and managed clusters for the following cloud providers: AWS, Azure, OpenStack, Equinix Metal with public or private networking, and non-MOSK-based bare metal.

For the vSphere and MOSK-based (managed) deployments, support for Ubuntu 20.04 will be announced in one of the following Container Cloud releases.

Note

The management or regional bare metal cluster dedicated for managed clusters running MOSK is based on Ubuntu 20.04.

Caution

Upgrading from Ubuntu 18.04 to 20.04 on existing deployments is not supported.

Learn more

Container Cloud on top of MOSK Victoria with Tungsten Fabric¶

Implemented the capability to deploy Container Cloud management, regional, and managed clusters based on OpenStack Victoria with Tungsten Fabric networking on top of Mirantis OpenStack for Kubernetes (MOSK) Victoria with Tungsten Fabric.

Note

On the MOSK Victoria with Tungsten Fabric clusters of Container Cloud deployed before MOSK 22.3, Octavia enables a default security group for newly created load balancers. To change this configuration, refer to MOSK Operations Guide: Configure load balancing. To use the default security group, configure ingress rules.

EBS instead of NVMe as persistent storage for AWS-based nodes¶

Replaced the Non-Volatile Memory Express (NVMe) drive type with the Amazon Elastic Block Store (EBS) one as the persistent storage requirement for AWS-based nodes. This change prevents cluster nodes from becoming unusable after instances are stopped and NVMe drives are erased.

Previously, the /var/lib/docker Docker data was located on local NVMe SSDs by default. Now, this data is located on the same EBS volume drive as the operating system.

Manager nodes deletion on all cluster types¶

TechPreview

Implemented the capability to delete manager nodes with the purpose of replacement or recovery. Consider the following precautions:

Create a new manager machine to replace the deleted one as soon as possible. This is necessary since after a machine removal, the cluster has limited capabilities to tolerate faults. Deletion of manager machines is intended only for replacement or recovery of failed nodes.
You can delete a manager machine only if your cluster has at least two manager machines in the Ready state.
Do not delete more than one manager machine at once to prevent cluster failure and data loss.
For MOSK-based clusters, after a manager machine deletion, proceed with additional manual steps described in Mirantis OpenStack for Kubernetes Operations Guide: Replace a failed controller node.
For the Equinix Metal and bare metal providers:
- Ensure that the machine to delete is not a Ceph Monitor. If it is, migrate the Ceph Monitor to keep the odd number quorum of Ceph Monitors after the machine deletion. For details, see Migrate a Ceph Monitor before machine replacement.
- If you delete a machine on the regional cluster, refer to the known issue 23853 to complete the deletion.

For the sake of HA, limited a managed cluster size to have only an odd number of manager machines. In an even-sized cluster, an additional machine remains in the Pending state until an extra manager machine is added.

Learn more

Operations Guide: Delete a cluster machine

Custom values for node labels¶

Extended the use of node labels for all supported cloud providers with the ability to set custom values. Especially from the MOSK standpoint, this feature makes it easy to schedule overrides for OpenStack services using API. For example, now you can set the node-type label to define the node purpose such as hpc-compute, compute-lvm, or storage-ssd in its value.

The list of allowed node labels is located in the Cluster object status providerStatus.releaseRef.current.allowedNodeLabels field. Before or after a machine deployment, add the required label from the allowed node labels list with the corresponding value to spec.providerSpec.value.nodeLabels in machine.yaml.

Note

Due to the known issue 23002, it is not possible to set a custom value for a predefined node label using the Container Cloud web UI. For a workaround, refer to the issue description.

Machine pools¶

Introduced the MachinePool custom resource. A machine pool is a template that allows managing a set of machines with the same provider spec as a single unit. You can create different sets of machine pools with required specs during machines creation on a new or existing cluster using the Create machine wizard in the Container Cloud web UI. You can assign or unassign machines from a pool, if required. You can also increase or decrease replicas count. In case of replicas count increasing, new machines will be added automatically.

Automatic propagation of Salesforce configuration to all clusters¶

Implemented the capability to enable automatic propagation of the Salesforce configuration of your management cluster to the related regional and managed clusters using the autoSyncSalesForceConfig=true flag added to the Cluster object of the management cluster. This option allows for automatic update and sync of the Salesforce settings on all your clusters after you update your management cluster configuration.

You can also set custom settings for regional and managed clusters that always override automatically propagated Salesforce values.

Note

The capability to enable this option using the Container Cloud web UI will be announced in one of the following releases.

Learn more

Operations Guide: Automatically propagate Salesforce configuration to all clusters

Addressed issues¶

The following issues have been addressed in the Mirantis Container Cloud release 2.17.0 along with the Cluster releases 11.1.0 and 7.7.0:

Bare metal:
- [22563] Fixed the issue wherein a deployment of a bare metal node with an LVM volume on top of a mdadm-based raid10 failed during provisioning due to insufficient cleanup of RAID devices.
Equinix Metal:
- [22264] Fixed the issue wherein the KubeContainersCPUThrottlingHigh alerts for Equinix Metal and AWS deployments raised due to low default deployment limits set for Equinix Metal and AWS controller containers.
StackLight:
- [23006] Fixed the issue that caused StackLight endpoints to crash on start with the private key does not match public key error message.
- [22626] Fixed the issue that caused constant restarts of the kaas-exporter pod. Increased the memory for kaas-exporter requests and limits.
- [22337] Improved the certificate expiration alerts by enhancing the alert severities.
- [20856] Fixed the issue wherein variables values in the PostgreSQL Grafana dashboard were not calculated.
- [20855] Fixed the issue wherein the Cluster > Health panel showed N/A in the Elasticsearch Grafana dashboard.
Ceph:
- [19014] Updated the Rook Docker image and fixed the following security vulnerabilities:
LCM:
- [22341] Fixed the issue wherein the cordon-drain states were not removed after unsetting the maintenance mode for a machine.
Cluster health:
- [21494] Fixed the issue wherein controller pods were killed by OOM after a successful deployment of a management or regional cluster.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.17.0 including the Cluster releases 11.1.0 and 7.7.0.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

MKE
Bare metal
Equinix Metal
IAM

LCM
StackLight
Upgrade
Container Cloud web UI

MKE¶

[20651] A cluster deployment or update fails with not ready compose deployments¶

'not ready: deployments: kube-system/compose got 0/0 replicas, kube-system/compose-api
 got 0/0 replicas'
 ready: false
 type: Kubernetes

Workaround:

Disable Docker Content Trust (DCT):
1. Access the MKE web UI as admin.
2. Navigate to Admin > Admin Settings.
3. In the left navigation pane, click Docker Content Trust and disable it.
Restart the affected deployments such as calico-kube-controllers, compose, compose-api, coredns, and so on:
```
kubectl -n kube-system delete deployment <deploymentName>
```
Once done, the cluster deployment or update resumes.
Re-enable DCT.

Bare metal¶

[20736] Region deletion failure after regional deployment failure¶

If a baremetal-based regional cluster deployment fails before pivoting is done, the corresponding region deletion fails.

Workaround:

cluster
machine
baremetalhost
baremetalhostprofile
l2template
subnet
ipamhost
ipaddr

kubectl delete <objectName> -l kaas.mirantis.com/region=<regionName>

Warning

Do not use the same region name again after the regional cluster deployment failure since some objects that reference the region name may still exist.

Equinix Metal¶

[16379,23865] Cluster update fails with the FailedMount warning¶

Fixed in 2.19.0

An Equinix-based management or managed cluster fails to update with the FailedAttachVolume and FailedMount warnings.

Workaround:

Verify that the description of the pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
- <affectedProjectName> is the Container Cloud project name where the pods failed to run
- <affectedPodName> is a pod name that failed to run in this project
In the pod description, identify the node name where the pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the pod that fails to init to 0 replicas.

On every csi-rbdplugin pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state is Running.

IAM¶

[18331] Keycloak admin console menu disappears on ‘Add identity provider’ page¶

Fixed in 2.18.0

During configuration of an identity provider SAML using the Add identity provider menu of the Keycloak admin console, the page style breaks as well as the Save and Cancel buttons disappear.

Workaround:

Log in to the Keycloak admin console.
In the sidebar menu, switch to the Master realm.
Navigate to Realm Settings > Themes.
In the Admin Console Theme drop-down menu, select keycloak.
Click Save and refresh the browser window to apply the changes.

LCM¶

[23853] Replacement of a regional master node fails on bare metal and Equinix Metal¶

Fixed in 2.18.0

During replacement of a failed master node on regional clusters of the bare metal and Equinix Metal providers, the KaaSCephOperationRequest resource created to remove the failed node from the Ceph cluster is stuck with the Failed status and an error message in errorReason. For example:

status:
  removeStatus:
    osdRemoveStatus:
      errorReason: Timeout (30m0s) reached for waiting pg rebalance for osd 2
      status: Failed

The Failed status blocks the replacement of the failed master node.

Workaround:

On the management cluster, obtain metadata.name, metadata.namespace, and the spec section of KaaSCephOperationRequest being stuck:
```
kubectl get kaascephoperationrequest <kcorName> -o yaml
```
Replace <kcorName> with the name of KaaSCephOperationRequest that has the Failed status.
Create a new KaaSCephOperationRequest template and save it as .yaml. For example, kcor-stuck-regional.yaml.
```
apiVersion: kaas.mirantis.com/v1alpha1
kind: KaaSCephOperationRequest
metadata:
 name: <newKcorName>
 namespace: <kcorNamespace>
spec: <kcorSpec>
```
- <newKcorName>
 Name of new KaaSCephOperationRequest that differs from the failed one. Usually a failed KaaSCephOperationRequest resource is called delete-request-for-<masterMachineName>. Therefore, you can name the new resource as delete-request-for-<masterMachineName>-new.
- <kcorNamespace>
 Namespace of the failed KaaSCephOperationRequest resource.
- <kcorSpec>
 Spec of the failed KaaSCephOperationRequest resource.
Apply the created template to the management cluster. For example:
```
kubectl apply -f kcor-stuck-regional.yaml
```
Remove the failed KaaSCephOperationRequest resource from the management cluster:
```
kubectl delete kaascephoperationrequest <kcorName>
```
Replace <kcorName> with the name of KaaSCephOperationRequest that has the Failed status.

StackLight¶

[20876] StackLight pods get stuck with the ‘NodeAffinity failed’ error¶

Note

Moving forward, the workaround for this issue will be moved from Release Notes to Operations Guide: Troubleshoot StackLight.

The issue does not affect the StackLight services, all required StackLight pods migrate successfully except extra pods that are created and stuck during pod migration.

As a workaround, remove the stuck pods:

kubectl --kubeconfig <managedClusterKubeconfig> -n stacklight delete pod <stuckPodName>

Upgrade¶

[21810] Upgrade to Cluster releases 5.22.0 and 7.5.0 may get stuck¶

Affects Ubuntu-based clusters deployed after Feb 10, 2022

To verify that the cluster is affected:

Log in to the Container Cloud web UI.
In the Clusters tab, capture the RELEASE and AGE values of the required Ubuntu-based cluster. If the values match the ones from the issue description, the cluster may be affected.
Using SSH, log in to the manager or worker node that got stuck while applying the Deploy state and identify the containerd package version:
```
containerd --version
```
If the version is 1.5.9, the cluster is affected.

In /var/log/lcm/runners/<nodeName>/deploy/, verify whether the Ansible deployment logs contain the following errors that indicate that the cluster is affected:

The following packages will be upgraded:
  docker-ee docker-ee-cli
The following packages will be DOWNGRADED:
  containerd.io

STDERR:
E: Packages were downgraded and -y was used without --allow-downgrades.

Workaround:

Warning

Using SSH, log in to the first affected node and install containerd 1.5.8:

apt-get install containerd.io=1.5.8-1 -y --allow-downgrades --allow-change-held-packages

Wait for Ansible to reconcile. The node should become Ready in several minutes.
Wait for the next node of the cluster to get stuck on the Deploy phase with the Ansible log errors. Only after that, apply the steps above on the next node.
Patch the remaining nodes one-by-one using the steps above.

Container Cloud web UI¶

[24075] Ubuntu 20.04 does not display for AWS and Equinix Metal managed clusters¶

Fixed in 2.18.0

During creation of a machine for AWS or Equinix Metal provider with public networking, the Ubuntu 20.04 option does not display in the drop-down list of operating systems in the Container Cloud UI. Only Ubuntu 18.04 displays in the list.

Workaround:

Identify the parent management or regional cluster of the affected managed cluster located in the same region.

For example, if the affected managed cluster was deployed in region-one, identify its parent cluster by running:
```
kubectl --kubeconfig <pathToManagementClusterKubeconfig> -n default get cluster -l kaas.mirantis.com/region=region-one
```
Replace region-one with the corresponding value.

Example of system response:
```
NAME AGE
test-cluster 19d
```

Modify the related management or regional Cluster object with the correct values for the credentials-controller Helm releases:

kubectl --kubeconfig <pathToManagementClusterKubeconfig> -n default edit cluster <managementOrRegionalClusterName>

In the system response, the editor displays the current state of the cluster. Find the spec.providerSpec.value.kaas.regional section.

Example of the regional section in the Cluster object:

spec:
  providerSpec:
    value:
      kaas:
        regional:
        - provider: aws
          helmReleases:
          - name: aws-credentials-controller
            values:
              region: region-one
              ...
        - provider: equinixmetal
          ...

For the aws and equinixmetal providers (if available), modify the credentials-controller values as follows:

Warning

Do not overwrite existing values. For example, if one of Helm releases already has region: region-one, do not modify or remove it.

For aws-credentials-controller:

values:
  config:
    allowedAMIs:
    -
      - name: name
        values:
          - "ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20211129"
      - name: owner-id
        values:
         - "099720109477"

For equinixmetal-credentials-controller:

values:
  config:
    allowedOperatingSystems:
    - distro: ubuntu
      version: 20.04

If the aws-credentials-controller or equinixmetal-credentials-controller Helm releases are missing in the spec.providerSpec.value.kaas.regional section or the helmReleases array is missing for the corresponding provider, add the releases with the overwritten values.

Example of the helmReleases array for AWS:

- provider: aws
  helmReleases:
  - name: aws-credentials-controller
    values:
      config:
        allowedAMIs:
        -
          - name: name
            values:
              - "ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20211129"
          - name: owner-id
            values:
             - "099720109477"
   ...

Example of the helmReleases array for Equinix Metal:

- provider: equinixmetal
  helmReleases:
  - name: equinixmetal-credentials-controller
    values:
      config:
        allowedOperatingSystems:
        - distro: ubuntu
          version: 20.04

Wait for approximately 2 minutes for the AWS and/or Equinix credentials-controller to be restarted.
Log out and log in again to the Container Cloud web UI.
Restart the machine addition procedure.

Warning

After Container Cloud is upgraded to 2.18.0, remove the values added during the workaround application from the Cluster object.

[23002] Inability to set a custom value for a predefined node label¶

Fixed in 7.11.0, 11.5.0 and 12.5.0

During machine creation using the Container Cloud web UI, a custom value for a node label cannot be set.

As a workaround, manually add the value to spec.providerSpec.value.nodeLabels in machine.yaml.

[249] A newly created project does not display in the Container Cloud web UI¶

Affects only Container Cloud 2.18.0 and earlier

Components versions¶

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.17.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

**Container Cloud release components versions**¶
Component	Application/Service	Version
AWS ^Updated	aws-provider	1.30.6
	aws-credentials-controller	1.30.6
Azure ^Updated	azure-provider	1.30.6
	azure-credentials-controller	1.30.6
Bare metal	ambassador	1.20.1-alpine
	baremetal-operator ^Updated	6.1.4
	baremetal-public-api ^Updated	6.1.4
	baremetal-provider ^Updated	1.30.6
	baremetal-resource-controller	base-focal-20220128182941
	ironic ^Updated	victoria-bionic-20220328060019
	ironic-operator ^Updated	base-focal-20220310095139
	kaas-ipam ^Updated	base-focal-20220310095439
	keepalived	2.1.5
	local-volume-provisioner	2.5.0-mcp
	mariadb	10.4.17-bionic-20220113085105
IAM	iam	2.4.14
	iam-controller ^Updated	1.30.6
	keycloak	15.0.2
Container Cloud	admission-controller ^Updated	1.30.6
	agent-controller ^Updated	1.30.6
	byo-credentials-controller ^Updated	1.30.6
	byo-provider ^Updated	1.30.6
	ceph-kcc-controller ^New	1.30.6
	cert-manager ^Updated	1.30.6
	client-certificate-controller ^Updated	1.30.6
	event-controller ^Updated	1.30.6
	golang	1.17.6
	kaas-public-api ^Updated	1.30.6
	kaas-exporter ^Updated	1.30.6
	kaas-ui ^Updated	1.30.9
	lcm-controller ^Updated	0.3.0-230-gdc7efe1c
	license-controller ^Updated	1.30.6
	machinepool-controller ^New	1.30.6
	mcc-cache ^Updated	1.30.6
	portforward-controller ^Updated	1.30.6
	proxy-controller ^Updated	1.30.6
	rbac-controller ^Updated	1.30.6
	release-controller ^Updated	1.30.8
	rhellicense-controller ^Updated	1.30.6
	scope-controller ^Updated	1.30.6
	squid-proxy	0.0.1-6
	user-controller ^Updated	1.30.6
Equinix Metal	equinix-provider ^Updated	1.30.6
	equinix-credentials-controller ^Updated	1.30.6
	keepalived	2.1.5
OpenStack ^Updated	openstack-provider	1.30.6
	os-credentials-controller	1.30.6
VMware vSphere	vsphere-provider ^Updated	1.30.6
	vsphere-credentials-controller ^Updated	1.30.6
	keepalived	2.1.5

Artifacts¶

This section lists the components artifacts of the Mirantis Container Cloud release 2.17.0.

Bare metal artifacts
Core artifacts
IAM artifacts

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶
Artifact	Component	Path
Binaries	baremetal-operator ^Updated	https://binary.mirantis.com/bm/helm/baremetal-operator-6.1.4.tgz
	baremetal-public-api ^Updated	https://binary.mirantis.com/bm/helm/baremetal-public-api-6.1.4.tgz
	ironic-python-agent.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-victoria-focal-debug-20220208120746
	ironic-python-agent.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-victoria-focal-debug-20220208120746
	kaas-ipam ^Updated	https://binary.mirantis.com/bm/helm/kaas-ipam-6.1.4.tgz
	local-volume-provisioner	https://binary.mirantis.com/bm/helm/local-volume-provisioner-2.5.0-mcp.tgz
	provisioning_ansible ^Updated	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-104-6e2e82c.tgz
	target ubuntu system	https://binary.mirantis.com/bm/bin/efi/ubuntu/tgz-bionic-20210622161844
Docker images	ambassador	mirantis.azurecr.io/general/external/docker.io/library/nginx:1.20.1-alpine
	baremetal-operator	mirantis.azurecr.io/bm/baremetal-operator:base-focal-20220208045851
	baremetal-resource-controller	mirantis.azurecr.io/bm/baremetal-resource-controller:base-focal-20220128182941
	dynamic_ipxe ^Updated	mirantis.azurecr.io/bm/dnsmasq/dynamic-ipxe:base-focal-20220310100410
	dnsmasq	mirantis.azurecr.io/general/dnsmasq:focal-20210617094827
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:victoria-bionic-20220328060019
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:victoria-bionic-20220328060019
	ironic-operator ^Updated	mirantis.azurecr.io/bm/ironic-operator:base-focal-20220310095139
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20210608113804
	kaas-ipam ^Updated	mirantis.azurecr.io/bm/kaas-ipam:base-focal-20220310095439
	mariadb	mirantis.azurecr.io/general/mariadb:10.4.17-bionic-20220113085105
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.14.0-1-g8725814
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-focal-20220128103433

Core artifacts¶
Artifact	Component	Paths
Bootstrap tarball ^Updated	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.30.6.tar.gz
	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.30.6.tar.gz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.30.6.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.30.6.tgz
	aws-credentials-controller	https://binary.mirantis.com/core/helm/aws-credentials-controller-1.30.6.tgz
	aws-provider	https://binary.mirantis.com/core/helm/aws-provider-1.30.6.tgz
	azure-credentials-controller	https://binary.mirantis.com/core/helm/azure-credentials-controller-1.30.6.tgz
	azure-provider	https://binary.mirantis.com/core/helm/azure-provider-1.30.6.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.30.6.tgz
	byo-credentials-controller	https://binary.mirantis.com/core/helm/byo-credentials-controller-1.30.6.tgz
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.30.6.tgz
	ceph-kcc-controller ^New	https://binary.mirantis.com/core/helm/ceph-kcc-controller-1.30.6.tgz
	cert-manager	https://binary.mirantis.com/core/helm/cert-manager-1.30.6.tgz
	client-certificate-controller	https://binary.mirantis.com/core/helm/client-certificate-controller-1.30.6.tgz
	equinix-credentials-controller	https://binary.mirantis.com/core/helm/equinix-credentials-controller-1.30.6.tgz
	equinix-provider	https://binary.mirantis.com/core/helm/equinix-provider-1.30.6.tgz
	equinixmetalv2-provider	https://binary.mirantis.com/core/helm/equinixmetalv2-provider-1.30.6.tgz
	event-controller	https://binary.mirantis.com/core/helm/event-controller-1.30.6.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.30.6.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.30.6.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.30.6.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.30.6.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.30.6.tgz
	license-controller ^Updated	https://binary.mirantis.com/core/helm/license-controller-1.30.6.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.30.6.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.30.6.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.30.6.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.30.6.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.30.6.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.30.6.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.30.8.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.30.6.tgz
	scope-controller	http://binary.mirantis.com/core/helm/scope-controller-1.30.6.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.30.6.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.30.6.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.30.6.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.30.6.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.30.6
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.30.6
	aws-cluster-api-controller ^Updated	mirantis.azurecr.io/core/aws-cluster-api-controller:1.30.6
	aws-credentials-controller ^Updated	mirantis.azurecr.io/core/aws-credentials-controller:1.30.6
	azure-cluster-api-controller ^Updated	mirantis.azurecr.io/core/azure-cluster-api-controller:1.30.6
	azure-credentials-controller ^Updated	mirantis.azurecr.io/core/azure-credentials-controller:1.30.6
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.30.6
	byo-credentials-controller ^Updated	mirantis.azurecr.io/core/byo-credentials-controller:1.30.6
	ceph-kcc-controller ^New	mirantis.azurecr.io/core/ceph-kcc-controller:v1.30.6
	cert-manager-controller ^Updated	mirantis.azurecr.io/core/external/cert-manager-controller:v1.6.1
	client-certificate-controller ^Updated	mirantis.azurecr.io/core/client-certificate-controller:1.30.6
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.30.6
	cluster-api-provider-equinix ^Updated	mirantis.azurecr.io/core/cluster-api-provider-equinix:1.30.6
	equinix-credentials-controller ^Updated	mirantis.azurecr.io/core/equinix-credentials-controller:1.30.6
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.30.6
	haproxy	mirantis.azurecr.io/lcm/mcc-haproxy:v0.12.0-8-g6fabf1c
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.30.6
	kproxy ^Updated	mirantis.azurecr.io/lcm/kproxy:1.30.6
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:v0.3.0-230-gdc7efe1c
	license-controller ^Updated	mirantis.azurecr.io/core/license-controller:1.30.6
	mcc-keepalived	mirantis.azurecr.io/lcm/mcc-keepalived:v0.14.0-1-g8725814
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.30.6
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.30.6
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.30.6
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.30.6
	registry	mirantis.azurecr.io/lcm/registry:2.7.1
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.30.8
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.30.6
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.30.6
	squid-proxy	mirantis.azurecr.io/core/squid-proxy:0.0.1-6
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-api-controller:1.30.6
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.30.6
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.30.6

IAM artifacts¶
Artifact	Component	Path
Binaries	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	iamctl-linux ^Updated	http://binary.mirantis.com/iam/bin/iamctl-0.5.5-linux
	iamctl-darwin ^Updated	http://binary.mirantis.com/iam/bin/iamctl-0.5.5-darwin
	iamctl-windows ^Updated	http://binary.mirantis.com/iam/bin/iamctl-0.5.5-windows
Helm charts	iam	http://binary.mirantis.com/iam/helm/iam-2.4.14.tgz
	iam-proxy	http://binary.mirantis.com/iam/helm/iam-proxy-0.2.12.tgz
	keycloak_proxy ^Updated	http://binary.mirantis.com/core/helm/keycloak_proxy-1.30.9.tgz
Docker images	api ^Deprecated	mirantis.azurecr.io/iam/api:0.5.5
	auxiliary	mirantis.azurecr.io/iam/auxiliary:0.5.5
	kubernetes-entrypoint	mirantis.azurecr.io/iam/external/kubernetes-entrypoint:v0.3.1
	mariadb	mirantis.azurecr.io/general/mariadb:10.4.16-bionic-20201105025052
	keycloak	mirantis.azurecr.io/iam/keycloak:0.5.4
	keycloak-gatekeeper	mirantis.azurecr.io/iam/keycloak-gatekeeper:7.1.3-2

See also

2.16.1¶

The Mirantis Container Cloud GA release 2.16.1 is based on 2.16.0 and:

Introduces support for the Cluster release 8.6.0 that is based on the Cluster release 7.6.0 and represents Mirantis OpenStack for Kubernetes (MOSK) 22.2. This Cluster release is based on the updated version of Mirantis Kubernetes Engine 3.4.7 with Kubernetes 1.20 and Mirantis Container Runtime 20.10.8.
Supports the latest Cluster releases 7.6.0 and 11.0.0.
Does not support new deployments based on the deprecated Cluster releases 8.5.0, 7.5.0, 6.20.0, and 5.22.0 that were deprecated in 2.16.0.

For details about the Container Cloud release 2.16.1, refer to its parent release 2.16.0:

Caution

See also

2.16.0¶

The Mirantis Container Cloud GA release 2.16.0:

Introduces support for the Cluster release 11.0.0 for managed clusters that is based on Mirantis Container Runtime 20.10.8 and the updated version of Mirantis Kubernetes Engine 3.5.1 with Kubernetes 1.21.
Introduces support for the Cluster release 7.6.0 for all types of clusters that is based on Mirantis Container Runtime 20.10.8 and the updated version of Mirantis Kubernetes Engine 3.4.7 with Kubernetes 1.20.
Supports the Cluster release 8.5.0 that is based on the Cluster release 7.5.0 and represents Mirantis OpenStack for Kubernetes (MOSK) 22.1.
Does not support greenfield deployments on deprecated Cluster releases 7.5.0, 6.20.0, and 5.22.0. Use the latest Cluster releases of the series instead.

Caution

Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.16.0.

Enhancements¶

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.16.0. For the list of enhancements in the Cluster releases 11.0.0 and 7.6.0 that are introduced by the Container Cloud release 2.16.0, see the Cluster releases (managed).

License management using the Container Cloud web UI
Scheduling of a management cluster upgrade using web UI
Automatic renewal of internal TLS certificates
Ubuntu 20.04 for greenfield bare metal managed clusters
Additional regional cluster on bare metal
MOSK on local RAID devices
Any interface name for bare metal LCM network
Keepalived for built-in load balancing in standalone containers
Reworked ‘Reconfigure’ phase of LCMMachine

License management using the Container Cloud web UI¶

Implemented a mechanism for the Container Cloud and MKE license update using the Container Cloud web UI. During the automatic license update, machines are not cordoned and drained and user workloads are not interrupted for all clusters starting from Cluster releases 7.6.0, 8.6.0, and 11.0.0. Therefore, after your management cluster upgrades to Container Cloud 2.16.0, make sure to update your managed clusters to the latest available Cluster releases.

Caution

Only the Container Cloud web UI users with the m:kaas@global-admin role can update the Container Cloud license.

Learn more

Scheduling of a management cluster upgrade using web UI¶

TechPreview

Implemented initial Technology Preview support for management cluster upgrade scheduling through the Container Cloud web UI. Also, added full support for management cluster upgrade scheduling through CLI.

Learn more

Schedule Mirantis Container Cloud updates

Automatic renewal of internal TLS certificates¶

Implemented automatic renewal of self-signed TLS certificates for internal Container Cloud services that are generated and managed by the Container Cloud provider.

Note

Custom certificates still require manual renewal. If applicable, the information about expiring custom certificates is available in the Container Cloud web UI.

Learn more

Configure TLS certificates for cluster applications

Ubuntu 20.04 for greenfield bare metal managed clusters¶

TechPreview

Implemented initial Technology Preview support for Ubuntu 20.04 (Focal Fossa) on bare metal non-MOSK-based greenfield deployments of managed clusters. Now, you can optionally deploy Kubernetes machines with Ubuntu 20.04 on bare metal hosts. By default, Ubuntu 18.04 is used.

Caution

Upgrading to Ubuntu 20.04 on existing deployments initially created before Container Cloud 2.16.0 is not supported.

Note

Support for Ubuntu 20.04 on MOSK-based Cluster releases will be added in one of the following Container Cloud releases.

Learn more

Create a machine using CLI

Additional regional cluster on bare metal¶

Extended the regional clusters support by implementing the ability to deploy an additional regional cluster on bare metal. This provides an ability to create baremetal-based managed clusters in bare metal regions in parallel with managed clusters of other private-based regional clusters within a single Container Cloud deployment.

MOSK on local RAID devices¶

TechPreview

Implemented the initial Technology Preview support for Mirantis OpenStack for Kubernetes (MOSK) deployment on local software-based Redundant Array of Independent Disks (RAID) devices to withstand failure of one device at a time. The feature is available in the Cluster release 8.5.0 after the Container Cloud upgrade to 2.16.0.

Using a custom bare metal host profile, you can configure and create an mdadm-based software RAID device of type raid10 if you have an even number of devices available on your servers. At least four storage devices are required for such RAID device.

Learn more

Create an mdadm software RAID level 10 (raid10)

Any interface name for bare metal LCM network¶

Implemented the ability to use any interface name instead of the k8s-lcm bridge for the LCM network traffic on a bare metal cluster. The Subnet objects for the LCM network must have the ipam/SVC-k8s-lcm label. For details, see Service labels and their life cycle.

Learn more

Keepalived for built-in load balancing in standalone containers¶

For the Container Cloud managed clusters that are based on vSphere, Equinix Metal, or bare metal, moved Keepalived for the built-in load balancer to run in standalone Docker containers managed by systemd as a service. This change ensures version consistency of crucial infrastructure services and reduces dependency on a host operating system version and configuration.

Learn more

Built-in load balancing

Reworked ‘Reconfigure’ phase of LCMMachine¶

Reworked the Reconfigure phase applicable to LCMMachine that now can apply to all nodes. This phase runs after the Deploy phase to apply stateItems that relate to this phase without affecting workloads running on the machine.

Learn more

LCM Controller

Addressed issues¶

The following issues have been addressed in the Mirantis Container Cloud release 2.16.0 along with the Cluster releases 11.0.0 and 7.6.0:

Bare metal:
- [15989] Fixed the issue wherein removal of a bare metal-based management cluster failed with a timeout.
- [20189] Fixed the issue with the Container Cloud web UI reporting a successful upgrade of a baremetal-based management cluster while running the previous release.
OpenStack:
- [20992] Fixed the issue that caused inability to deploy an OpenStack-based managed cluster if DVR was enabled.
- [20549] Fixed the CVE-2021-3520 security vulnerability in the cinder-csi-plugin image Docker image.
Equinix Metal:
- [20467] Fixed the issue that caused deployment of an Equinix Metal based management cluster with private networking to fail with the following error message during the Ironic deployment:
```
0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.
```
- [21324] Fixed the issue wherein the bare metal host was trying to configure an Equinix node as UEFI even for nodes with UEFI disabled.
- [21326] Fixed the issue wherein the Ironic agent could not properly determine which disk will be the first disk on the node. As a result, some Equinix servers failed to boot from the proper disk.
- [21338] Fixed the issue wherein some Equinix servers were configured in BIOS to always boot from PXE, which caused the operation system to fail to start from disk after provisioning.
StackLight:
- [21646] Adjusted the kaas-exporter resource requests and limits to avoid issues with the kaas-exporter container being occassionally throttled and OOMKilled, preventing the Container Cloud metrics gathering.
- [20591] Adjusted the RAM usage limit and disabled indices monitoring for prometheus-es-exporter to avoid prometheus-es-exporter pod crash looping due to low memory issues.
- [17493] Fixed the following security vulnerabilities in the fluentd and spilo Docker images:
Ceph:
- [20745] Fixed the issue wherein namespace deletion failed after removal of a managed cluster.
- [7073] Fixed the issue with inability to automatically remove a Ceph node when removing a worker node.
IAM:
- [20157] Fixed the CVE-2019-20916 security vulnerability in IAM.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.16.0 including the Cluster releases 11.0.0 and 7.6.0.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

MKE
Equinix Metal
Bare metal

IAM
StackLight
LCM

Upgrade
Container Cloud web UI
Cluster health

MKE¶

[20651] A cluster deployment or update fails with not ready compose deployments¶

'not ready: deployments: kube-system/compose got 0/0 replicas, kube-system/compose-api
 got 0/0 replicas'
 ready: false
 type: Kubernetes

Workaround:

Disable Docker Content Trust (DCT):
1. Access the MKE web UI as admin.
2. Navigate to Admin > Admin Settings.
3. In the left navigation pane, click Docker Content Trust and disable it.
Restart the affected deployments such as calico-kube-controllers, compose, compose-api, coredns, and so on:
```
kubectl -n kube-system delete deployment <deploymentName>
```
Once done, the cluster deployment or update resumes.
Re-enable DCT.

Equinix Metal¶

[22264] KubeContainersCPUThrottlingHigh alerts for Equinix and AWS deployments¶

Fixed in 2.17.0

The default deployment limits for Equinix and AWS controller containers set to 400m may be lower than the consumed amount of resources leading to KubeContainersCPUThrottlingHigh alerts in StackLight.

As a workaround, increase the default resource limits for the affected equinix-controllers or aws-controllers to 700m. For example:

kubectl edit deployment -n kaas aws-controllers

spec:
...
  resources:
    limits:
      cpu: 700m
      ...

[16379,23865] Cluster update fails with the FailedMount warning¶

Fixed in 2.19.0

An Equinix-based management or managed cluster fails to update with the FailedAttachVolume and FailedMount warnings.

Workaround:

Verify that the description of the pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
- <affectedProjectName> is the Container Cloud project name where the pods failed to run
- <affectedPodName> is a pod name that failed to run in this project
In the pod description, identify the node name where the pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the pod that fails to init to 0 replicas.

On every csi-rbdplugin pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state is Running.

Bare metal¶

[20736] Region deletion failure after regional deployment failure¶

If a baremetal-based regional cluster deployment fails before pivoting is done, the corresponding region deletion fails.

Workaround:

cluster
machine
baremetalhost
baremetalhostprofile
l2template
subnet
ipamhost
ipaddr

kubectl delete <objectName> -l kaas.mirantis.com/region=<regionName>

Warning

Do not use the same region name again after the regional cluster deployment failure since some objects that reference the region name may still exist.

[22563] Failure to deploy a bare metal node with RAID 1¶

Fixed in 2.17.0

Deployment of a bare metal node with an mdadm-based raid10 with LVM enabled fails during provisioning due to insufficient cleanup of RAID devices.

Workaround:

Boot the affected node from any LiveCD, preferably Ubuntu.
Obtain details about the mdadm RAID devices:
```
sudo mdadm --detail --scan --verbose
```
Stop all mdadm RAID devices listed in the output of the above command. For example:
```
sudo mdadm --stop /dev/md0
```
Clean up the metadata on partitions with the mdadm RAID device(s) enabled. For example:
```
sudo mdadm --zero-superblock /dev/sda1
```
In the above example, replace /dev/sda1 with partitions listed in the output of the command provided in the step 2.

[17792] Full preflight fails with a timeout waiting for BareMetalHost¶

If you run bootstrap.sh preflight with KAAS_BM_FULL_PREFLIGHT=true, the script fails with the following message:

preflight check failed: preflight full check failed: \
error waiting for BareMetalHosts to power on: \
timed out waiting for the condition

Workaround:

Unset full preflight using the unset KAAS_BM_FULL_PREFLIGHT environment variable.
Rerun bootstrap.sh preflight that executes fast preflight instead.

IAM¶

[18331] Keycloak admin console menu disappears on ‘Add identity provider’ page¶

Fixed in 2.18.0

During configuration of an identity provider SAML using the Add identity provider menu of the Keycloak admin console, the page style breaks as well as the Save and Cancel buttons disappear.

Workaround:

Log in to the Keycloak admin console.
In the sidebar menu, switch to the Master realm.
Navigate to Realm Settings > Themes.
In the Admin Console Theme drop-down menu, select keycloak.
Click Save and refresh the browser window to apply the changes.

StackLight¶

[20876] StackLight pods get stuck with the ‘NodeAffinity failed’ error¶

Note

Moving forward, the workaround for this issue will be moved from Release Notes to Operations Guide: Troubleshoot StackLight.

The issue does not affect the StackLight services, all required StackLight pods migrate successfully except extra pods that are created and stuck during pod migration.

As a workaround, remove the stuck pods:

kubectl --kubeconfig <managedClusterKubeconfig> -n stacklight delete pod <stuckPodName>

[23006] StackLight endpoint crashes on start: private key does not match public key¶

Fixed in 2.17.0

In rare cases, StackLight applications may receive the wrong TLS certificates, which prevents them to start correctly.

As a workaround, delete the old secret for the affected StackLight component. For example, for iam-proxy-alerta:

kubectl -n stacklight delete secret iam-proxy-alerta-tls-certs

LCM¶

[22341] The cordon-drain states are not removed after maintenance mode is unset¶

Fixed in 2.17.0

The cordon-drain states are not removed after the maintenance mode is unset for a machine. This issue may occur due to the maintenance transition being stuck on the NodeWorkloadLock object.

Workaround:

Select from the following options:

Disable the maintenance mode on the affected cluster as described in Enable cluster and machine maintenance mode.

Edit LCMClusterState in the spec section by setting value to "false":

kubectl edit lcmclusterstates -n <projectName> <LCMCLusterStateName>

apiVersion: lcm.mirantis.com/v1alpha1
kind: LCMClusterState
metadata:
  ...
spec:
  ...
  value: "false"

Upgrade¶

[21810] Upgrade to Cluster releases 5.22.0 and 7.5.0 may get stuck¶

Affects Ubuntu-based clusters deployed after Feb 10, 2022

To verify that the cluster is affected:

Log in to the Container Cloud web UI.
In the Clusters tab, capture the RELEASE and AGE values of the required Ubuntu-based cluster. If the values match the ones from the issue description, the cluster may be affected.
Using SSH, log in to the manager or worker node that got stuck while applying the Deploy state and identify the containerd package version:
```
containerd --version
```
If the version is 1.5.9, the cluster is affected.

In /var/log/lcm/runners/<nodeName>/deploy/, verify whether the Ansible deployment logs contain the following errors that indicate that the cluster is affected:

The following packages will be upgraded:
  docker-ee docker-ee-cli
The following packages will be DOWNGRADED:
  containerd.io

STDERR:
E: Packages were downgraded and -y was used without --allow-downgrades.

Workaround:

Warning

Using SSH, log in to the first affected node and install containerd 1.5.8:

apt-get install containerd.io=1.5.8-1 -y --allow-downgrades --allow-change-held-packages

Wait for Ansible to reconcile. The node should become Ready in several minutes.
Wait for the next node of the cluster to get stuck on the Deploy phase with the Ansible log errors. Only after that, apply the steps above on the next node.
Patch the remaining nodes one-by-one using the steps above.

Container Cloud web UI¶

[249] A newly created project does not display in the Container Cloud web UI¶

Affects only Container Cloud 2.18.0 and earlier

Cluster health¶

[21494] Controller pods are OOMkilled after deployment¶

Fixed in 2.17.0

After a successful deployment of a management or regional cluster, controller pods may be OOMkilled and get stuck in CrashLoopBackOff state due to incorrect memory limits.

Workaround:

Increase memory resources limits on the affected Deployment:

Open the affected Deployment configuration for editing:

kubectl --kubeconfig <mgmtOrRegionalKubeconfig> -n kaas edit deployment <deploymentName>

Update the value of spec.template.spec.containers.resources.limits by 100-200 Mi. For example:

spec:
  template:
    spec:
      containers:
      - ...
        resources:
          limits:
            cpu: "3"
            memory: 500Mi
          requests:
            cpu: "1"
            memory: 300Mi

Components versions¶

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.16.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

**Container Cloud release components versions**¶
Component	Application/Service	Version
AWS ^Updated	aws-provider	1.29.6
	aws-credentials-controller	1.29.6
Azure ^Updated	azure-provider	1.29.6
	azure-credentials-controller	1.29.6
Bare metal	ambassador	1.20.1-alpine
	baremetal-operator ^Updated	6.1.2
	baremetal-public-api ^Updated	6.1.3
	baremetal-provider ^Updated	1.29.9
	baremetal-resource-controller ^Updated	base-focal-20220128182941
	ironic ^Updated	victoria-bionic-20220208100053
	ironic-operator ^Updated	base-focal-20220217095047
	kaas-ipam ^Updated	base-focal-20220131093130
	keepalived ^Updated	2.1.5
	local-volume-provisioner	2.5.0-mcp
	mariadb ^Updated	10.4.17-bionic-20220113085105
IAM	iam ^Updated	2.4.14
	iam-controller ^Updated	1.29.6
	keycloak	15.0.2
Container Cloud	admission-controller ^Updated	1.29.7
	agent-controller ^Updated	1.29.6
	byo-credentials-controller ^Updated	1.29.6
	byo-provider ^Updated	1.29.6
	cert-manager ^New	1.29.6
	client-certificate-controller ^New	1.29.6
	event-controller ^New	1.29.6
	golang ^Updated	1.17.6
	kaas-public-api ^Updated	1.29.6
	kaas-exporter ^Updated	1.29.6
	kaas-ui ^Updated	1.29.6
	lcm-controller ^Updated	0.3.0-187-gba894556
	license-controller ^New	1.29.6
	mcc-cache ^Updated	1.29.6
	portforward-controller ^Updated	1.29.6
	proxy-controller ^Updated	1.29.6
	rbac-controller ^Updated	1.29.6
	release-controller ^Updated	1.29.7
	rhellicense-controller ^Updated	1.29.6
	scope-controller ^Updated	1.29.6
	squid-proxy	0.0.1-6
	user-controller ^Updated	1.29.6
Equinix Metal ^Updated	equinix-provider	1.29.6
	equinix-credentials-controller	1.29.6
	keepalived	2.1.5
OpenStack ^Updated	openstack-provider	1.29.6
	os-credentials-controller	1.29.6
VMware vSphere ^Updated	vsphere-provider	1.29.6
	vsphere-credentials-controller	1.29.6
	keepalived	2.1.5

Artifacts¶

This section lists the components artifacts of the Mirantis Container Cloud release 2.16.0.

Bare metal artifacts
Core artifacts
IAM artifacts

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶
Artifact	Component	Path
Binaries	baremetal-operator ^Updated	https://binary.mirantis.com/bm/helm/baremetal-operator-6.1.2.tgz
	baremetal-public-api ^Updated	https://binary.mirantis.com/bm/helm/baremetal-public-api-6.1.3.tgz
	ironic-python-agent-bionic.kernel ^Removed	Replaced with `ironic-python-agent.kernel`
	ironic-python-agent-bionic.initramfs ^Removed	Replaced with `ironic-python-agent.initramfs`
	ironic-python-agent.initramfs ^New	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-victoria-focal-debug-20220208120746
	ironic-python-agent.kernel ^New	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-victoria-focal-debug-20220208120746
	kaas-ipam ^Updated	https://binary.mirantis.com/bm/helm/kaas-ipam-6.1.2.tgz
	local-volume-provisioner	https://binary.mirantis.com/bm/helm/local-volume-provisioner-2.5.0-mcp.tgz
	provisioning_ansible ^Updated	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-102-08af94e.tgz
	target ubuntu system	https://binary.mirantis.com/bm/bin/efi/ubuntu/tgz-bionic-20210622161844
Docker images	ambassador	mirantis.azurecr.io/general/external/docker.io/library/nginx:1.20.1-alpine
	baremetal-operator ^Updated	mirantis.azurecr.io/bm/baremetal-operator:base-focal-20220208045851
	baremetal-resource-controller ^Updated	mirantis.azurecr.io/bm/baremetal-resource-controller:base-focal-20220128182941
	dynamic_ipxe ^New	mirantis.azurecr.io/bm/dnsmasq/dynamic-ipxe:base-focal-20220126144549
	dnsmasq ^Updated	mirantis.azurecr.io/general/dnsmasq:focal-20210617094827
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:victoria-bionic-20220208100053
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:victoria-bionic-20220208100053
	ironic-operator ^Updated	mirantis.azurecr.io/bm/ironic-operator:base-focal-20220217095047
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20210608113804
	kaas-ipam ^Updated	mirantis.azurecr.io/bm/kaas-ipam:base-focal-20220131093130
	mariadb ^Updated	mirantis.azurecr.io/general/mariadb:10.4.17-bionic-20220113085105
	mcc-keepalived ^New	mirantis.azurecr.io/lcm/mcc-keepalived:v0.14.0-1-g8725814
	syslog-ng ^Updated	mirantis.azurecr.io/bm/syslog-ng:base-focal-20220128103433

Core artifacts¶
Artifact	Component	Paths
Bootstrap tarball ^Updated	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.29.6.tar.gz
	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.29.6.tar.gz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.29.7.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.29.6.tgz
	aws-credentials-controller	https://binary.mirantis.com/core/helm/aws-credentials-controller-1.29.6.tgz
	aws-provider	https://binary.mirantis.com/core/helm/aws-provider-1.29.6.tgz
	azure-credentials-controller	https://binary.mirantis.com/core/helm/azure-credentials-controller-1.29.6.tgz
	azure-provider	https://binary.mirantis.com/core/helm/azure-provider-1.29.6.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.29.9.tgz
	byo-credentials-controller	https://binary.mirantis.com/core/helm/byo-credentials-controller-1.29.6.tgz
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.29.6.tgz
	cert-manager ^New	https://binary.mirantis.com/core/helm/cert-manager-1.29.6.tgz
	client-certificate-controller ^New	https://binary.mirantis.com/core/helm/client-certificate-controller-1.29.6.tgz
	equinix-credentials-controller	https://binary.mirantis.com/core/helm/equinix-credentials-controller-1.29.6.tgz
	equinix-provider	https://binary.mirantis.com/core/helm/equinix-provider-1.29.6.tgz
	equinixmetalv2-provider	https://binary.mirantis.com/core/helm/equinixmetalv2-provider-1.29.6.tgz
	event-controller ^New	https://binary.mirantis.com/core/helm/event-controller-1.29.6.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.29.6.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.29.6.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.29.6.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.29.6.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.29.6.tgz
	license-controller ^New	https://binary.mirantis.com/core/helm/license-controller-1.29.6.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.29.6.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.29.6.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.29.6.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.29.6.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.29.6.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.29.6.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.29.7.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.29.6.tgz
	scope-controller	http://binary.mirantis.com/core/helm/scope-controller-1.29.6.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.29.6.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.29.6.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.29.6.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.29.6.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.29.7
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.29.6
	aws-cluster-api-controller ^Updated	mirantis.azurecr.io/core/aws-cluster-api-controller:1.29.6
	aws-credentials-controller ^Updated	mirantis.azurecr.io/core/aws-credentials-controller:1.29.6
	azure-cluster-api-controller ^Updated	mirantis.azurecr.io/core/azure-cluster-api-controller:1.29.6
	azure-credentials-controller ^Updated	mirantis.azurecr.io/core/azure-credentials-controller:1.29.6
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.29.6
	byo-credentials-controller ^Updated	mirantis.azurecr.io/core/byo-credentials-controller:1.29.6
	cert-manager-controller ^New	mirantis.azurecr.io/core/external/cert-manager-controller:v1.6.1
	client-certificate-controller ^New	mirantis.azurecr.io/core/client-certificate-controller:1.29.6
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.29.6
	cluster-api-provider-equinix ^Updated	mirantis.azurecr.io/core/cluster-api-provider-equinix:1.29.6
	equinix-credentials-controller ^Updated	mirantis.azurecr.io/core/equinix-credentials-controller:1.29.6
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.29.6
	haproxy	mirantis.azurecr.io/lcm/mcc-haproxy:v0.12.0-8-g6fabf1c
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.29.6
	kproxy ^Updated	mirantis.azurecr.io/lcm/kproxy:1.29.6
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:v0.3.0-187-gba894556
	license-controller ^New	mirantis.azurecr.io/core/license-controller:1.29.6
	mcc-keepalived ^New	mirantis.azurecr.io/lcm/mcc-keepalived:v0.14.0-1-g8725814
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.29.6
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.29.6
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.29.6
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.29.6
	registry	mirantis.azurecr.io/lcm/registry:2.7.1
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.29.7
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.29.6
	scope-controller ^Updated	mirantis.azurecr.io/core/scope-controller:1.29.6
	squid-proxy	mirantis.azurecr.io/core/squid-proxy:0.0.1-6
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-api-controller:1.29.6
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.29.6
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.29.6

IAM artifacts¶
Artifact	Component	Path
Binaries	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	iamctl-linux ^Updated	http://binary.mirantis.com/iam/bin/iamctl-0.5.5-linux
	iamctl-darwin ^Updated	http://binary.mirantis.com/iam/bin/iamctl-0.5.5-darwin
	iamctl-windows ^Updated	http://binary.mirantis.com/iam/bin/iamctl-0.5.5-windows
Helm charts	iam ^Updated	http://binary.mirantis.com/iam/helm/iam-2.4.14.tgz
	iam-proxy ^Updated	http://binary.mirantis.com/iam/helm/iam-proxy-0.2.12.tgz
	keycloak_proxy ^Updated	http://binary.mirantis.com/core/helm/keycloak_proxy-1.29.8.tgz
Docker images	api ^Deprecated	mirantis.azurecr.io/iam/api:0.5.5
	auxiliary ^Updated	mirantis.azurecr.io/iam/auxiliary:0.5.5
	kubernetes-entrypoint	mirantis.azurecr.io/iam/external/kubernetes-entrypoint:v0.3.1
	mariadb	mirantis.azurecr.io/general/mariadb:10.4.16-bionic-20201105025052
	keycloak	mirantis.azurecr.io/iam/keycloak:0.5.4
	keycloak-gatekeeper	mirantis.azurecr.io/iam/keycloak-gatekeeper:7.1.3-2

See also

2.15.1¶

The Mirantis Container Cloud GA release 2.15.1 is based on 2.15.0 and:

Introduces support for the Cluster release 8.5.0 that is based on the Cluster release 7.5.0 and represents Mirantis OpenStack for Kubernetes (MOSK) 22.1. This Cluster release is based on Mirantis Kubernetes Engine 3.4.6 with Kubernetes 1.20 and Mirantis Container Runtime 20.10.8.
Supports the latest Cluster releases 7.5.0 and 5.22.0.
Does not support new deployments based on the Cluster releases 7.4.0 and 5.21.0 that were deprecated in 2.15.0.

For details about the Container Cloud release 2.15.1, refer to its parent release 2.15.0:

Caution

See also

2.15.0¶

The Mirantis Container Cloud GA release 2.15.0:

Introduces support for the Cluster release 7.5.0 that is based on Mirantis Container Runtime 20.10.8 and the updated version of Mirantis Kubernetes Engine 3.4.6 with Kubernetes 1.20.
Introduces support for the Cluster release 5.22.0 that is based on the updated version of Mirantis Kubernetes Engine 3.3.13 with Kubernetes 1.18 and Mirantis Container Runtime 20.10.8.
Supports the Cluster release 6.20.0 that is based on the Cluster release 5.20.0 and represents Mirantis OpenStack for Kubernetes (MOS) 21.6.
Does not support greenfield deployments on deprecated Cluster releases 7.4.0, 6.19.0, and 5.21.0. Use the latest Cluster releases of the series instead.

Caution

Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.15.0.

Enhancements¶

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.15.0. For the list of enhancements in the Cluster releases 7.5.0 and 5.22.0 that are supported by the Container Cloud release 2.15.0, see the Cluster releases (managed).

Automatic upgrade of bare metal host operating system during cluster update
Dedicated subnet for externally accessible Kubernetes API endpoint
HAProxy instead of NGINX for vSphere, Equinix Metal, and bare metal providers
Additional regional cluster on Equinix Metal with private networking
Scheduled Container Cloud auto-upgrade
Cluster and machine maintenance mode
Improvements for monitoring of machine deployment live status
Deprecation of iam-api and IAM CLI
Switch of Ceph Helm releases from v2 to v3
Documentation enhancements

Automatic upgrade of bare metal host operating system during cluster update¶

Introduced automatic upgrade of Ubuntu 18.04 packages on the bare metal hosts during a management or managed cluster update.

Mirantis Container Cloud uses life cycle management tools to update the operating system packages on the bare metal hosts. Container Cloud may also trigger restart of the bare metal hosts to apply the updates, when applicable.

Warning

During managed cluster update to the latest Cluster releases available in Container Cloud 2.15.0, hosts are restarted to apply the latest supported Ubuntu 18.04 packages and update kernel to version 5.4.0-90.101.

If Ceph is installed in the cluster, the Container Cloud orchestration securely pauses the Ceph OSDs on the node before restart. This allows avoiding degradation of the storage service.

Learn more

Dedicated subnet for externally accessible Kubernetes API endpoint¶

TechPreview

Implemented a capability to add a dedicated subnet for the externally accessible Kubernetes API endpoint of a baremetal-based managed cluster.

Learn more

Create subnets for a managed cluster using CLI

HAProxy instead of NGINX for vSphere, Equinix Metal, and bare metal providers¶

Implemented a health check mechanism to verify target server availability by reworking the high availability setup for the Container Cloud manager nodes of the vSphere, Equinix Metal, and bare metal providers to use HAProxy instead of NGINX. This change affects only the Ansible part. HAproxy deploys as a container managed directly by containerd.

Learn more

Additional regional cluster on Equinix Metal with private networking¶

Extended the regional clusters support by implementing the capability to deploy an additional regional cluster on Equinix Metal with private networking. This provides the capability to create managed clusters in the Equinix Metal regions with private networking in parallel with managed clusters of other supported providers within a single Container Cloud deployment.

Scheduled Container Cloud auto-upgrade¶

TechPreview

Introduced the initial Technology Preview support for a scheduled Container Cloud auto-upgrade using the MCCUpgrade object named mcc-upgrade in Kubernetes API.

An Operator can delay or reschedule Container Cloud auto-upgrade that allows:

Blocking Container Cloud upgrade process for up to 7 days from the current date and up to 30 days from the latest Container Cloud release
Limiting hours and weekdays when Container Cloud upgrade can run

Caution

Only the management cluster admin has access to the MCCUpgrade object. You must use kubeconfig generated during the management cluster bootstrap to access this object.

Note

Scheduling of the Container Cloud auto-upgrade using the Container Cloud web UI will be implemented in one of the following releases.

Learn more

Cluster and machine maintenance mode¶

Implemented the maintenance mode for management and managed clusters and machines to prepare workloads for maintenance operations.

To enable maintenance mode on a machine, first enable maintenance mode on a related cluster.
To disable maintenance mode on a cluster, first disable maintenance mode on all machines of the cluster.

Warning

Cluster upgrades and configuration changes (except of the SSH keys setting) are unavailable while a cluster is under maintenance. Make sure you disable maintenance mode on the cluster after maintenance is complete.

Learn more

Operatoins Guide: Enable cluster and machine maintenance mode

Improvements for monitoring of machine deployment live status¶

Implemented the following improvements to the live status of a machine deployment that you can monitor using the Container Cloud web UI:

Increased the events coverage
Added information about cordon and drain (if a node is being cordoned, drained, or uncordoned) to the Kubelet and Swarm machine components statuses.

These improvements are implemented for all supported Container Cloud providers.

Deprecation of iam-api and IAM CLI¶

Deprecated the iam-api service and IAM CLI (the iamctl command). The logic of the iam-api service required for Container Cloud is moved to scope-controller. The iam-api service is used by IAM CLI only to manage users and permissions. Instead of IAM CLI, Mirantis recommends using the Keycloak web UI to perform necessary IAM operations.

The iam-api service and IAM CLI will be removed in one of the following Container Cloud releases.

Switch of Ceph Helm releases from v2 to v3¶

Upgraded the Ceph Helm releases in the ClusterRelease object from v2 to v3. Switching of the remaining OpenStack Helm releases for Mirantis OpenStack for Kubernetes to v3 will be implemented in one of the following Container Cloud releases.

Documentation enhancements¶

On top of continuous improvements delivered to the existing Container Cloud guides, added the following procedures:

Expand IP addresses capacity in an existing cluster for the bare metal provider
Calculate target ratio for Ceph pools

Addressed issues¶

The following issues have been addressed in the Mirantis Container Cloud release 2.15.0 along with the Cluster releases 7.5.0 and 5.22.0:

vSphere:
- [19737] Fixed the issue with the vSphere VM template build hanging with an empty kickstart file on the vSphere deployments with the RHEL 8.4 seed node.
- [19468] Fixed the issue with the ‘Failed to remove finalizer from machine’ error during cluster deletion if a RHEL license is removed before the related managed cluster was deleted.
IAM:
- [5025] Updated the Keycloak version from 12.0.0 to 15.0.2 to fix the CVE-2020-2757.
- [21024][Custom certificates] Fixed the issue with the readiness check failure during addition of a custom certificate for Keycloak that hung with the failed to wait for OIDC certificate to be updated timeout warning.
StackLight:
- [20193] Updated the Grafana Docker image from 8.2.2 to 8.2.7 to fix the high-severity CVE-2021-43798.
- [18933] Fixed the issue with the Alerta pods failing to pass the readiness check even if Patroni, the Alerta backend, operated correctly.
- [19682] Fixed the issue with the Prometheus web UI URLs in notifications sent to Salesforce using the HTTP protocol instead of HTTPS on deployments with TLS enabled for IAM.
Ceph:
- [19645] Fixed the issue with the Ceph OSD removal request failure during the Processing stage.
- [19574] Fixed the issue with the Ceph OSD removal not cleaning up the device used for multiple OSDs.
- [20298] Fixed the issue with spec validation failing during creation of KaaSCephOperationRequest.
- [20355] Fixed the issue with KaaSCephOperationRequest being cached after recreation with the same name, specified in metadata.name, as the previous KaaSCephOperationRequest CR. The issue caused no removal to be performed upon applying the new KaaSCephOperationRequest CR.
Bare metal:
- [19786] Fixed the issue with managed cluster deployment failing on long-running management clusters with BareMetalHost being stuck in the Preparing state and the ironic-conductor and ironic-api pods reporting the not enough disk space error due to the dnsmasq-dhcpd logs overflow.
Upgrade:
- [20459] Fixed the issue with failure to upgrade a management or regional cluster originally deployed using the Container Cloud release earlier than 2.8.0. The failure occurred during Ansible update if a machine contained /usr/local/share/ca-certificates/mcc.crt, which was either empty or invalid.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.15.0 including the Cluster releases 7.5.0 and 5.22.0.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

MKE
Hardware-related
Equinix Metal
Bare metal
IAM
LCM
Monitoring
Upgrade
Container Cloud web UI

MKE¶

[20651] A cluster deployment or update fails with not ready compose deployments¶

'not ready: deployments: kube-system/compose got 0/0 replicas, kube-system/compose-api
 got 0/0 replicas'
 ready: false
 type: Kubernetes

Workaround:

Disable Docker Content Trust (DCT):
1. Access the MKE web UI as admin.
2. Navigate to Admin > Admin Settings.
3. In the left navigation pane, click Docker Content Trust and disable it.
Restart the affected deployments such as calico-kube-controllers, compose, compose-api, coredns, and so on:
```
kubectl -n kube-system delete deployment <deploymentName>
```
Once done, the cluster deployment or update resumes.
Re-enable DCT.

Equinix Metal¶

[20467] Failure to deploy an Equinix Metal based management cluster¶

Fixed in 2.16.0

Deployment of an Equinix Metal based management cluster with private networking may fail with the following error message during the Ironic deployment. The issue is caused by csi-rbdplugin provisioner pods that got stuck.

0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.

The workaround is to restart the csi-rbdplugin provisioner pods:

kubectl -n rook-ceph delete pod -l app=csi-rbdplugin-provisioner

Bare metal¶

[20745] Namespace deletion failure after managed cluster removal¶

Fixed in 2.16.0

After removal of a managed cluster, the namespace is not deleted due to KaaSCephOperationRequest CRs blocking the deletion. The workaround is to manually remove finalizers and delete the KaaSCephOperationRequest CRs.

Workaround:

Remove finalizers from all KaaSCephOperationRequest resources:

kubectl -n <managed-ns> get kaascephoperationrequest -o name | xargs -I % kubectl -n <managed-ns> patch % -p '{"metadata":{"finalizers":{}}}' --type=merge

Delete all KaaSCephOperationRequest resources:

kubectl -n <managed-ns> delete kaascephoperationrequest --all

[17792] Full preflight fails with a timeout waiting for BareMetalHost¶

If you run bootstrap.sh preflight with KAAS_BM_FULL_PREFLIGHT=true, the script fails with the following message:

preflight check failed: preflight full check failed: \
error waiting for BareMetalHosts to power on: \
timed out waiting for the condition

Workaround:

Unset full preflight using the unset KAAS_BM_FULL_PREFLIGHT environment variable.
Rerun bootstrap.sh preflight that executes fast preflight instead.

IAM¶

[18331] Keycloak admin console menu disappears on ‘Add identity provider’ page¶

Fixed in 2.18.0

During configuration of an identity provider SAML using the Add identity provider menu of the Keycloak admin console, the page style breaks as well as the Save and Cancel buttons disappear.

Workaround:

Log in to the Keycloak admin console.
In the sidebar menu, switch to the Master realm.
Navigate to Realm Settings > Themes.
In the Admin Console Theme drop-down menu, select keycloak.
Click Save and refresh the browser window to apply the changes.

LCM¶

[22341] The cordon-drain states are not removed after maintenance mode is unset¶

Fixed in 2.17.0

The cordon-drain states are not removed after the maintenance mode is unset for a machine. This issue may occur due to the maintenance transition being stuck on the NodeWorkloadLock object.

Workaround:

Select from the following options:

Disable the maintenance mode on the affected cluster as described in Enable cluster and machine maintenance mode.

Edit LCMClusterState in the spec section by setting value to "false":

kubectl edit lcmclusterstates -n <projectName> <LCMCLusterStateName>

apiVersion: lcm.mirantis.com/v1alpha1
kind: LCMClusterState
metadata:
  ...
spec:
  ...
  value: "false"

Monitoring¶

[20876] StackLight pods get stuck with the ‘NodeAffinity failed’ error¶

Note

Moving forward, the workaround for this issue will be moved from Release Notes to Operations Guide: Troubleshoot StackLight.

The issue does not affect the StackLight services, all required StackLight pods migrate successfully except extra pods that are created and stuck during pod migration.

As a workaround, remove the stuck pods:

kubectl --kubeconfig <managedClusterKubeconfig> -n stacklight delete pod <stuckPodName>

[21646] The kaas-exporter container is periodically throttled and OOMKilled¶

Fixed in 2.16.0

On the highly loaded clusters, the kaas-exporter resource limits for CPU and RAM are lower than the consumed amount of resources. As a result, the kaas-exporter container is periodically throttled and OOMKilled preventing the Container Cloud metrics gathering.

As a workaround, increase the default resource limits for kaas-exporter in the Cluster object of the management cluster. For example:

spec:
  ...
  providerSpec:
    ...
    value:
      ...
      kaas:
        management:
          helmReleases:
          ...
          - name: kaas-exporter
            values:
              resources:
                limits:
                  cpu: 100m
                  memory: 200Mi

Upgrade¶

[21810] Upgrade to Cluster releases 5.22.0 and 7.5.0 may get stuck¶

Affects Ubuntu-based clusters deployed after Feb 10, 2022

To verify that the cluster is affected:

Log in to the Container Cloud web UI.
In the Clusters tab, capture the RELEASE and AGE values of the required Ubuntu-based cluster. If the values match the ones from the issue description, the cluster may be affected.
Using SSH, log in to the manager or worker node that got stuck while applying the Deploy state and identify the containerd package version:
```
containerd --version
```
If the version is 1.5.9, the cluster is affected.

In /var/log/lcm/runners/<nodeName>/deploy/, verify whether the Ansible deployment logs contain the following errors that indicate that the cluster is affected:

The following packages will be upgraded:
  docker-ee docker-ee-cli
The following packages will be DOWNGRADED:
  containerd.io

STDERR:
E: Packages were downgraded and -y was used without --allow-downgrades.

Workaround:

Warning

Using SSH, log in to the first affected node and install containerd 1.5.8:

apt-get install containerd.io=1.5.8-1 -y --allow-downgrades --allow-change-held-packages

Wait for Ansible to reconcile. The node should become Ready in several minutes.
Wait for the next node of the cluster to get stuck on the Deploy phase with the Ansible log errors. Only after that, apply the steps above on the next node.
Patch the remaining nodes one-by-one using the steps above.

[20189] Container Cloud web UI reports upgrade while running previous release¶

Fixed in 2.16.0

Under certain conditions, the upgrade of the baremetal-based management cluster may get stuck even though the Container Cloud web UI reports a successful upgrade. The issue is caused by inconsistent metadata in IPAM that prevents automatic allocation of the Ceph network. It happens when IPAddr objects associated with the management cluster nodes refer to a non-existent Subnet object by the resource UID.

To verify whether the cluster is affected:

Inspect the baremetal-provider logs:

kubectl -n kaas logs deployments/baremetal-provider

If the logs contain the following entries, the cluster may be affected:

Ceph public network address validation failed for cluster default/kaas-mgmt: invalid address '0.0.0.0/0' \

Ceph cluster network address validation failed for cluster default/kaas-mgmt: invalid address '0.0.0.0/0' \

'default/kaas-mgmt' cluster nodes internal (LCM) IP addresses: 10.64.96.171,10.64.96.172,10.64.96.173 \

failed to configure ceph network for cluster default/kaas-mgmt: \
Ceph network addresses auto-assignment error: validation failed for Ceph network addresses: \
error parsing address '': invalid CIDR address:

Empty values of the network parameters in the last entry indicate that the provider cannot locate the Subnet object based on the data from the IPAddr object.

Note

In the logs, capture the internal (LCM) IP addresses of the cluster nodes to use them later in this procedure.

Validate the network address used for Ceph by inspecting the MiraCeph object:
```
kubectl -n ceph-lcm-mirantis get miraceph -o yaml | egrep "^ +clusterNet:"
kubectl -n ceph-lcm-mirantis get miraceph -o yaml | egrep "^ +publicNet:"
```
In the system response, verify that the clusterNet and publicNet values do not contain the 0.0.0.0/0 range.

Example of the system response on the affected cluster:
```
clusterNet: 0.0.0.0/0

publicNet: 0.0.0.0/0
```

Workaround:

Add a label to the Subnet object:

Note

To obtain the correct name of the label, use one of the cluster nodes internal (LCM) IP addresses from the baremetal-provider logs.
1. Add SUBNETID as an environment variable to the IPAddr object. For example:
```
SUBNETID=$(kubectl get ipaddr -n default --selector=ipam/IP=10.64.96.171 -o custom-columns=":metadata.labels.ipam/SubnetID" | tr -d '\n')
```
2. Use the SUBNETID variable to restore the required label in the Subnet object:
```
kubectl -n default label subnet master-region-one ipam/UID-${SUBNETID}="1"
```
Verify that the cluster.sigs.k8s.io/cluster-name label exists for IPaddr objects:
```
kubectl -n default get ipaddr --show-labels|grep "cluster.sigs.k8s.io/cluster-name"
```
Skip the next step if all IPaddr objects corresponding to the management cluster nodes have this label.

Add the cluster.sigs.k8s.io/cluster-name label to IPaddr objects:

IPADDRNAMES=$(kubectl -n default get ipaddr -o custom-columns=":metadata.name")
for IP in $IPADDRNAMES; do kubectl -n default label ipaddr $IP cluster.sigs.k8s.io/cluster-name=<managementClusterName>; done

In the command above, substitute <managementClusterName> with the corresponding value.

[16379,23865] Cluster update fails with the FailedMount warning¶

Fixed in 2.19.0

An Equinix-based management or managed cluster fails to update with the FailedAttachVolume and FailedMount warnings.

Workaround:

Verify that the description of the pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
- <affectedProjectName> is the Container Cloud project name where the pods failed to run
- <affectedPodName> is a pod name that failed to run in this project
In the pod description, identify the node name where the pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the pod that fails to init to 0 replicas.

On every csi-rbdplugin pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state is Running.

Container Cloud web UI¶

[249] A newly created project does not display in the Container Cloud web UI¶

Affects only Container Cloud 2.18.0 and earlier

Components versions¶

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.15.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

**Container Cloud release components versions**¶
Component	Application/Service	Version
AWS ^Updated	aws-provider	1.28.7
	aws-credentials-controller	1.28.7
Azure ^Updated	azure-provider	1.28.7
	azure-credentials-controller	1.28.7
Bare metal	ambassador	1.20.1-alpine
	baremetal-operator ^Updated	6.0.4
	baremetal-public-api ^Updated	6.0.4
	baremetal-provider ^Updated	1.28.7
	baremetal-resource-controller ^Updated	base-bionic-20211224163705
	ironic ^Updated	victoria-bionic-20211213142623
	ironic-operator	base-bionic-20210930105000
	kaas-ipam ^Updated	base-bionic-20211213150212
	local-volume-provisioner	2.5.0-mcp
	mariadb	10.4.17-bionic-20210617085111
IAM	iam	2.4.10
	iam-controller ^Updated	1.28.7
	keycloak ^Updated	15.0.2
Container Cloud ^Updated	admission-controller	1.28.7 (1.28.18 for 2.15.1)
	agent-controller	1.28.7
	byo-credentials-controller	1.28.7
	byo-provider	1.28.7
	kaas-public-api	1.28.7
	kaas-exporter	1.28.7
	kaas-ui	1.28.8
	lcm-controller	0.3.0-132-g83a348fa
	mcc-cache	1.28.7
	portforward-controller	1.28.12
	proxy-controller	1.28.7
	rbac-controller	1.28.7
	release-controller	1.28.7
	rhellicense-controller	1.28.7
	scope-controller ^New	1.28.7
	squid-proxy	0.0.1-6
	user-controller	1.28.7
Equinix Metal ^Updated	equinix-provider	1.28.11
	equinix-credentials-controller	1.28.7
OpenStack ^Updated	openstack-provider	1.28.7
	os-credentials-controller	1.28.7
VMware vSphere ^Updated	vsphere-provider	1.28.7
	vsphere-credentials-controller	1.28.7

Artifacts¶

This section lists the components artifacts of the Mirantis Container Cloud release 2.15.0.

Bare metal artifacts
Core artifacts
IAM artifacts

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶
Artifact	Component	Path
Binaries	baremetal-operator ^Updated	https://binary.mirantis.com/bm/helm/baremetal-operator-6.0.4.tgz
	baremetal-public-api ^Updated	https://binary.mirantis.com/bm/helm/baremetal-public-api-6.0.4.tgz
	ironic-python-agent-bionic.kernel ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-victoria-bionic-5.4-debug-20211126120723
	ironic-python-agent-bionic.initramfs ^Updated	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-victoria-bionic-5.4-debug-20211126120723
	kaas-ipam ^Updated	https://binary.mirantis.com/bm/helm/kaas-ipam-6.0.4.tgz
	local-volume-provisioner	https://binary.mirantis.com/bm/helm/local-volume-provisioner-2.5.0-mcp.tgz
	provisioning_ansible ^Updated	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-88-02063c4.tgz
	target ubuntu system	https://binary.mirantis.com/bm/bin/efi/ubuntu/tgz-bionic-20210622161844
Docker images	ambassador	mirantis.azurecr.io/general/external/docker.io/library/nginx:1.20.1-alpine
	baremetal-operator	mirantis.azurecr.io/bm/baremetal-operator:base-bionic-20211005112459
	baremetal-resource-controller ^Updated	mirantis.azurecr.io/bm/baremetal-resource-controller:base-bionic-20211224163705
	dnsmasq	mirantis.azurecr.io/general/dnsmasq:focal-20210617094827
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:victoria-bionic-20211213142623
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:victoria-bionic-20211213142623
	ironic-operator	mirantis.azurecr.io/bm/ironic-operator:base-bionic-20210930105000
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20210608113804
	kaas-ipam ^Updated	mirantis.azurecr.io/bm/kaas-ipam:base-bionic-20211213150212
	mariadb	mirantis.azurecr.io/general/mariadb:10.4.17-bionic-20210617085111
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-bionic-20210617094817

Core artifacts¶
Artifact	Component	Paths
Bootstrap tarball ^Updated	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.28.7.tar.gz
	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.28.7.tar.gz
Helm charts ^Updated	admission-controller 0	https://binary.mirantis.com/core/helm/admission-controller-1.28.7.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.28.7.tgz
	aws-credentials-controller	https://binary.mirantis.com/core/helm/aws-credentials-controller-1.28.7.tgz
	aws-provider	https://binary.mirantis.com/core/helm/aws-provider-1.28.7.tgz
	azure-credentials-controller	https://binary.mirantis.com/core/helm/azure-credentials-controller-1.28.7.tgz
	azure-provider	https://binary.mirantis.com/core/helm/azure-provider-1.28.7.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.28.7.tgz
	byo-credentials-controller	https://binary.mirantis.com/core/helm/byo-credentials-controller-1.28.7.tgz
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.28.7.tgz
	equinix-credentials-controller	https://binary.mirantis.com/core/helm/equinix-credentials-controller-1.28.7.tgz
	equinix-provider	https://binary.mirantis.com/core/helm/equinix-provider-1.28.11.tgz
	equinixmetalv2-provider	https://binary.mirantis.com/core/helm/equinixmetalv2-provider-1.28.7.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.28.7.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.28.7.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.28.7.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.27.8.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.28.7.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.28.7.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.28.7.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.28.7.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.28.7.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.28.7.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.28.7.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.28.7.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.28.7.tgz
	scope-controller ^New	http://binary.mirantis.com/core/helm/scope-controller-1.28.7.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.28.7.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.28.7.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.28.7.tgz
	user-controller	https://binary.mirantis.com/core/helm/user-controller-1.28.7.tgz
Docker images	admission-controller 0 ^Updated	mirantis.azurecr.io/core/admission-controller:1.28.7
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.28.7
	aws-cluster-api-controller ^Updated	mirantis.azurecr.io/core/aws-cluster-api-controller:1.28.7
	aws-credentials-controller ^Updated	mirantis.azurecr.io/core/aws-credentials-controller:1.28.7
	azure-cluster-api-controller ^Updated	mirantis.azurecr.io/core/azure-cluster-api-controller:1.28.7
	azure-credentials-controller ^Updated	mirantis.azurecr.io/core/azure-credentials-controller:1.28.7
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.28.7
	byo-credentials-controller ^Updated	mirantis.azurecr.io/core/byo-credentials-controller:1.28.7
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.28.7
	cluster-api-provider-equinix ^Updated	mirantis.azurecr.io/core/cluster-api-provider-equinix:1.28.7
	equinix-credentials-controller ^Updated	mirantis.azurecr.io/core/equinix-credentials-controller:1.28.7
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.28.8
	haproxy	mirantis.azurecr.io/lcm/mcc-haproxy:v0.12.0-8-g6fabf1c
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.28.7
	kproxy ^Updated	mirantis.azurecr.io/lcm/kproxy:1.28.7
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:v0.3.0-132-g83a348fa
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.28.7
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.28.7
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.28.12
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.28.7
	registry	mirantis.azurecr.io/lcm/registry:2.7.1
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.28.7
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.28.7
	scope-controller ^New	mirantis.azurecr.io/core/scope-controller:1.28.7
	squid-proxy ^Updated	mirantis.azurecr.io/core/squid-proxy:0.0.1-6
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-api-controller:1.28.7
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.28.7
	user-controller ^Updated	mirantis.azurecr.io/core/user-controller:1.28.7

0(1,2): In Container Cloud 2.15.1, the version of admission-controller is 1.28.18.

IAM artifacts¶
Artifact	Component	Path
Binaries	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	iamctl-linux	http://binary.mirantis.com/iam/bin/iamctl-0.5.4-linux
	iamctl-darwin	http://binary.mirantis.com/iam/bin/iamctl-0.5.4-darwin
	iamctl-windows	http://binary.mirantis.com/iam/bin/iamctl-0.5.4-windows
Helm charts	iam	http://binary.mirantis.com/iam/helm/iam-2.4.10.tgz
	iam-proxy ^Updated	http://binary.mirantis.com/iam/helm/iam-proxy-0.2.10.tgz
	keycloak_proxy ^Updated	http://binary.mirantis.com/core/helm/keycloak_proxy-1.28.9.tgz
Docker images	api ^Deprecated	mirantis.azurecr.io/iam/api:0.5.4
	auxiliary	mirantis.azurecr.io/iam/auxiliary:0.5.4
	kubernetes-entrypoint	mirantis.azurecr.io/iam/external/kubernetes-entrypoint:v0.3.1
	mariadb	mirantis.azurecr.io/general/mariadb:10.4.16-bionic-20201105025052
	keycloak	mirantis.azurecr.io/iam/keycloak:0.5.4
	keycloak-gatekeeper	mirantis.azurecr.io/iam/keycloak-gatekeeper:7.1.3-2

See also

Releases delivered in 2020-2021¶

This section contains historical information on the unsupported Container Cloud releases delivered in 2020-2021. For the latest supported Container Cloud release, see Container Cloud releases.

Unsupported Container Cloud releases 2020-2021¶
Version	Release date	Supported Cluster releases	Summary
2.14.0	Dec 07, 2021	7.4.0 7.3.0 ^* 6.20.0 6.19.0 ^* 5.21.0 5.20.0 ^*	Equinix Metal provider: Support of the private networking infrastructure as Technology Preview OpenStack provider: Support of the community version of CentOS 7.9 Configuration of server metadata for machines in web UI vSphere provider: Initial RHEL 8.4 support ^TechPreview Configuration of RAM and CPU for machines in web UI Bare metal provider: Visualization of service mapping in the bare metal IpamHost object MKE support: Support matrix of MKE versions for cluster attachment MKE version update from 3.3.12 to 3.3.13 in the Cluster release 5.21.0 and from 3.4.5 to 3.4.6 in the Cluster release 7.4.0 IAM: User access management through the Container Cloud API or web UI Updated role naming used in Keycloak LCM: Switch of bare metal and StackLight Helm releases from v2 to v3 StackLight: Network interfaces monitoring Custom Prometheus recording rules Syslog packet size configuration Prometheus Relay configuration Ceph: Enhanced architecture Networks validation Automated Ceph OSD removal ^TechPreview Container Cloud web UI: The ‘Interface Guided Tour’ button in the Container Cloud web UI
2.13.1	Nov 11, 2021	6.20.0 6.19.0 ^*	Based on 2.13.0, this release introduces the Cluster release 6.20.0 that is based on 5.20.0 and supports Mirantis OpenStack for Kubernetes (MOS) 21.6. For the list of Cluster releases 7.x and 5.x that are supported by 2.13.1 as well as for its features with addressed and known issues, refer to the parent release 2.13.0.
2.13.0	Oct 28, 2021	7.3.0 7.2.0 ^* 6.19.0 6.18.0 ^* 5.20.0 5.19.0 ^*	Configuration of multiple DHCP ranges for bare metal clusters Updated RAM requirements for management and regional clusters Improvements to StackLight alerting Support for Telegraf 1.20.0 Documentation: How to renew the Container Cloud and MKE licenses
2.12.0	Oct 5, 2021	7.2.0 7.1.0 ^* 6.19.0 6.18.0 ^* 5.19.0 5.18.0 ^*	General availability of the Microsoft Azure cloud provider Support for the Container Cloud deployment on top of MOS Victoria TLS for all Container Cloud endpoints LVM or mdadm RAID support for bare metal provisioning `Preparing` state of a bare metal host Migration of `iam-proxy` from Louketo Proxy to OAuth2 Proxy Backup configuration for a MariaDB database on a management cluster Renaming of the Container Cloud binary from `kaas` to `container-cloud` MCR version update to 20.10.6 MKE version update to 3.4.5 for the Cluster release 7.2.0 and to 3.3.12 for Cluster releases 5.19.0, 6.19.0 Ceph: Integration of the Ceph maintenance to the common upgrade procedure Ceph RADOS Gateway tolerations StackLight: Short names for Kubernetes nodes in Grafana dashboards Improvements to StackLight alerting Logs-based metrics in StackLight Documentation: How to back up and restore an OpenStack or AWS-based management cluster
2.11.0	August 31, 2021	7.1.0 7.0.0 ^* 6.18.0 6.16.0 ^* 5.18.0 5.17.0 ^* 5.11.0 ^**	Technology Preview support for the Microsoft Azure cloud provider RHEL 7.9 bootstrap node for the vSphere-based provider Validation labels for the vSphere-based VM templates Automatic migration of Docker data and LVP volumes to NVMe on AWS clusters Switch of core Helm releases from v2 to v3 Bond interfaces for baremetal-based management clusters Bare metal advanced configuration using web UI Equinix Metal capacity labels for machines in web UI Ceph: Support for Ceph Octopus Hyperconverged Ceph improvement Ceph cluster status improvements Ceph Manager modules StackLight: StackLight node labeling improvements StackLight log level severity setting in web UI Improvements to StackLight alerting Salesforce feed update Documentation: How to manually remove a Ceph OSD from a Ceph cluster How to update the Keycloak IP address on bare metal clusters
2.10.0	July 21, 2021	7.0.0 6.16.0 6.14.0 ^* 5.17.0 5.16.0 ^* 5.11.0 ^**	7.x Cluster release series with updated versions of MCR 20.10.5, MKE 3.4.0, and Kubernetes 1.20.1 Support of MKE 3.3.3 - 3.3.6 and 3.4.0 for cluster attachment Graceful MCR upgrade from 19.03.14 to 20.10.5 MKE logs gathering enhancements VMware vSphere provider: Initial CentOS support VMware vSphere provider RHEL 7.9 support for the VMware vSphere provider Removal of IAM and Keycloak IPs configuration Ability to add or configure proxy on existing clusters Command for creation of Keycloak users Improvements to StackLight alerting Log verbosity for StackLight components Documentation: How to move a Ceph Monitor daemon to another node Manage user roles through Keycloak
2.9.0	June 15, 2021	6.16.0 6.14.0 ^* 5.16.0 5.15.0 ^* 5.11.0 ^**	Equinix Metal provider Integration to Lens New bootstrap node for additional regional clusters TLS certificates for management cluster applications Default Keycloak authorization in Container Cloud web UI SSH keys management for `mcc-user` vSphere resources controller StackLight components upgrade Ceph: Multinetwork configuration TLS for public endpoints RBD mirroring support
2.8.0	May 18, 2021	6.14.0 6.12.0 ^* 5.15.0 5.14.0 ^* 5.11.0 ^**	Support for Keycloak 12.0 Ironic pod logs LoadBalancer and ProviderInstance monitoring for cluster and machine statuses Updated notification about outdated cluster version in web UI StackLight improvements: Notifications to Microsoft Teams Notifications to ServiceNow Log collection optimization Ceph improvements: Ceph default configuration options Capability to define specifications for multiple Ceph nodes using lists A number of new KaaSCephCluster configuration parameters Documentation enhancements: Ceph Monitors recovery Silencing of StackLight alerts
2.7.0	April 22, 2021	6.14.0 6.12.0 ^* 5.14.0 5.13.0 ^* 5.11.0 ^**	Full support for the VMware vSphere provider Universal SSH user Configuration of SSH keys on existing clusters using web UI Cluster and machines live statuses in web UI Enabling of proxy access using web UI for vSphere, AWS, and bare metal Log collection optimization in StackLight Ceph enhancements: Dedicated network for the Ceph distributed storage traffic Ceph Multisite configuration Documentation enhancements: Ceph disaster recovery procedure QuickStart guides
2.6.0	March 24, 2021	6.12.0 6.10.0 ^* 5.13.0 5.12.0 ^* 5.11.0 ^**	RHEL license activation using the activation key Support for VMware vSphere Distributed Switch VMware vSphere provider integration with IPAM controller Proxy support for all Container Cloud providers StackLight logging levels StackLight remote logging to syslog Hyperconverged Ceph Ceph `objectStorage` section in `KaasCephCluster` Ceph maintenance orchestration Updated documentation on the bare metal networking
2.5.0	March 1, 2021	6.12.0 6.10.0 ^* 5.12.0 5.11.0 ^*	Support for Mirantis Kubernetes Engine 3.3.6 Support for Mirantis OpenStack for Kubernetes 21.1 Proxy support for OpenStack and VMware vSphere providers NTP server configuration on regional clusters Optimized `ClusterRelease` upgrade process Dedicated network for external connection to the Kubernetes services on bare metal Ceph RADOS Gateway HA Ceph RADOS Gateway check box in Container Cloud web UI Ceph maintenance label Cerebro support for StackLight Proxy support for StackLight
2.4.0	February 2, 2021	6.10.0 6.8.1 ^* 5.11.0 5.10.0 ^*	Support for the updated version of Mirantis Container Runtime 19.03.14 Dedicated network for Kubernetes pods traffic on bare metal clusters Improvements for the feedback form in the Container Cloud web UI StackLight enhancements: Alert inhibition rules Integration between Grafana and Kibana New Telegraf alert `TelegrafGatherErrors` Configuration of Ironic Telegraf input plugin Automatically defined cluster ID
2.3.0	December 23, 2020	6.10.0 6.8.1 ^* 5.10.0 5.9.0 ^*	Support for Mirantis Kubernetes Engine 3.3.4 and Mirantis Container Runtime 19.03.13 Support for multiple host-specific L2 templates per a bare metal cluster Additional regional cluster on VMware vSphere Automated setup of a VM template for the VMware vSphere provider StackLight support for VMware vSphere Improvements in the Container Cloud logs collection
2.2.0	November 5, 2020	6.8.1 5.9.0 5.8.0	Support for VMware vSphere provider on RHEL Kernel parameters management through BareMetalHostProfile Support of multiple subnets per cluster Optimization of the Container Cloud logs collection Container Cloud API documentation for bare metal
2.1.0	October 19, 2020	5.8.0 5.7.0	Node labeling for machines AWS resources discovery in the Container Cloud web UI Credentials statuses for OpenStack and AWS in the Container Cloud web UI StackLight improvements: Grafana upgrade from version 6.6.2 to 7.1.5 Grafana Image Renderer pod to offload rendering of images from charts Grafana home dashboard improvements Splitting of the regional and management cluster function in StackLight telemetry to obtain aggregated metrics on the management cluster from regional and managed clusters Amendments to the StackLight alerts
2.0.0	September 16, 2020	5.7.0	First GA release of Container Cloud with the following key features: Container Cloud with Mirantis Kubernetes Engine (MKE) container clusters for the management plane Support for managed Container Cloud with MKE container clusters on top of the AWS, OpenStack, and bare metal cloud providers Support for attaching of the existing MKE standalone clusters Ceph as a Kubernetes storage provider for the bare metal use case Multi-region support for security and scalability IAM integration with MKE container clusters to provide SSO Logging, monitoring, and alerting tuned for MKE with data aggregation to the management cluster and telemetry sent to Mirantis

^** - the Cluster release supports only attachment of existing MKE 3.3.4 clusters. For the deployment of new or attachment of existing clusters based on other supported MKE versions, the latest available Cluster releases are used.

2.14.0¶

The Mirantis Container Cloud GA release 2.14.0:

Introduces support for the Cluster release 7.4.0 that is based on Mirantis Container Runtime 20.10.6 and the updated version of Mirantis Kubernetes Engine 3.4.6 with Kubernetes 1.20.
Introduces support for the Cluster release 5.21.0 that is based on the updated version of Mirantis Kubernetes Engine 3.3.13 with Kubernetes 1.18 and Mirantis Container Runtime 20.10.6.
Supports the Cluster release 6.20.0 that is based on the Cluster release 5.20.0 and represents Mirantis OpenStack for Kubernetes (MOS) 21.6.
Supports deprecated Cluster releases 5.20.0, 6.19.0, and 7.3.0 that will become unsupported in the following Container Cloud releases.

Caution

Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.14.0.

Enhancements¶

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.14.0. For the list of enhancements in the Cluster releases 7.4.0 and 5.21.0 that are supported by the Container Cloud release 2.14.0, see the Cluster releases (managed).

Support of the Equinix Metal provider with private networking
Support of the community CentOS 7.9 version for the OpenStack provider
Configuration of server metadata for OpenStack machines in web UI
Initial RHEL 8.4 support for the vSphere provider
Configuration of RAM and CPU for vSphere machines in web UI
Visualization of service mapping in the bare metal IpamHost object
Separation of PXE and management networks for bare metal clusters
User access management through the Container Cloud API or web UI
Support matrix of MKE versions for cluster attachment
The ‘Interface Guided Tour’ button in the Container Cloud web UI
Switch of bare metal and StackLight Helm releases from v2 to v3

Support of the Equinix Metal provider with private networking¶

TechPreview

Introduced the Technology Preview support of Container Cloud deployments that are based on the Equinix Metal infrastructure with private networking.

Private networks are required for the following use cases:

Connect the Container Cloud to the on-premises corporate networks without exposing it to the Internet. This can be required by corporate security policies.
Reduce ingress and egress bandwidth costs and the number of public IP addresses utilized by the deployment. Public IP addresses are a scarce and valuable resource, and Container Cloud should only expose the necessary services in that address space.
Testing and staging environments typically do not require accepting connections from the outside of the cluster. Such Container Cloud clusters should be isolated in private VLANs.

Caution

The feature is supported starting from the Cluster releases 7.4.0 and 5.21.0.

Note

Support of the regional clusters that are based on Equinix Metal with private networking will be announced in one of the following Container Cloud releases.

Support of the community CentOS 7.9 version for the OpenStack provider¶

Introduced support of the community version of the CentOS 7.9 operating system for the management, regional, and managed clusters machines deployed with the OpenStack provider. The following CentOS resources are used:

Latest upstream CentOS 7.9 image: CentOS-7-x86_64-GenericCloud-2009.qcow2
Latest CentOS 7.9 .yum repositories: mirror.centos.org

Configuration of server metadata for OpenStack machines in web UI¶

Implemented the possibility to specify the cloud-init metadata during the OpenStack machines creation through the Container Cloud web UI. Server metadata is a set of string key-value pairs that you can configure in the meta_data field of cloud-init.

Initial RHEL 8.4 support for the vSphere provider¶

TechPreview

Introduced the initial Technology Preview support of the RHEL 8.4 operating system for the vSphere-based management, regional, and managed clusters.

Caution

Deployment of a Container Cloud cluster based on both RHEL and CentOS operating systems or on mixed RHEL versions is not supported.

Configuration of RAM and CPU for vSphere machines in web UI¶

Implemented the possibility to configure the following settings during a vSphere machine creation using the Container Cloud web UI:

VM memory size that defaults to 16 GB
VM CPUs number that defaults to 8

Visualization of service mapping in the bare metal IpamHost object¶

Implemented the following amendments to the ipam/SVC-* labels to simplify visualization of service mapping in the bare metal IpamHost object:

All IP addresses allocated from the Subnet` object that has the ipam/SVC-* service labels defined will inherit those labels
The new ServiceMap field in IpamHost.Status contains information about which IPs and interfaces correspond to which Container Cloud services.

Learn more

Service labels and their life cycle

Separation of PXE and management networks for bare metal clusters¶

Added the capability to configure a dedicated PXE network that is separated from the management network on management or regional bare metal clusters. A separate PXE network allows isolating sensitive bare metal provisioning process from the end users. The users still have access to Container Cloud services, such as Keycloak, to authenticate workloads in managed clusters, such as Horizon in a Mirantis OpenStack for Kubernetes cluster.

Learn more

Separate PXE and management networks

User access management through the Container Cloud API or web UI¶

Implemented the capability to manage user access through the Container Cloud API or web UI by introducing the following objects to manage user role bindings:

IAMUser
IAMRole
IAMGlobalRoleBinding
IAMRoleBinding
IAMClusterRoleBinding

Also, updated the role naming used in Keycloak by introducing the following IAM roles with the possibility to upgrade the old-style role names with the new-style ones:

global-admin
bm-pool-operator
operator
user
stacklight-admin

Caution

User management for the MOSK m:os roles through API or web UI is on the final development stage and will be announced in one of the following Container Cloud releases. Meanwhile, continue managing these roles using Keycloak.
The possibility to manage the IAM*RoleBinding objects through the Container Cloud web UI is available for the global-admin role only. The possibility to manage project role bindings using the operator role will become available in one of the following Container Cloud releases.

Learn more

Support matrix of MKE versions for cluster attachment¶

Updated the matrix of supported MKE versions for cluster attachment to improve the upgrade and testing procedures:

Implemented separate Cluster release series to support 2 series of MKE versions for cluster attachment:
- Cluster release series 9.x for the 3.3.x version series
- Cluster release series 10.x for the 3.4.x version series
Added a requirement to update an existing MKE cluster to the latest available supported MKE version in a series to trigger the Container Cloud upgrade that allows updating its components, such as StackLight, to the latest versions.

When a new MKE version for cluster attachment is released in a series, the oldest supported version of the previous Container Cloud release is dropped.

The ‘Interface Guided Tour’ button in the Container Cloud web UI¶

Added the Interface Guided Tour button to the Container Cloud web UI Help section for a handy access to the guided tour that steps you through the web UI key features of the multi-cluster multi-cloud Container Cloud platform.

Switch of bare metal and StackLight Helm releases from v2 to v3¶

Upgraded the bare metal and StackLight Helm releases in the ClusterRelease and KaasRelease objects from v2 to v3. Switching of the remaining Ceph and OpenStack Helm releases to v3 will be implemented in one of the following Container Cloud releases.

Addressed issues¶

The following issues have been addressed in the Mirantis Container Cloud release 2.14.0 along with the Cluster releases 7.4.0 and 5.21.0.

[18429][StackLight] Increased the default resource requirements for Prometheus Elasticsearch Exporter to prevent the KubeContainersCPUThrottlingHigh firing too often.
[18879][Ceph] Fixed the issue with the RADOS Gateway (RGW) pod overriding the global CA bundle located at /etc/pki/tls/certs with an incorrect self-signed CA bundle during deployment of a Ceph cluster.
[9899][Upgrade] Fixed the issue with Helm releases getting stuck in the PENDING_UPGRADE state during a management or managed cluster upgrade.
[18708][LCM] Fixed the issue with the Pending state of machines during deployment of any Container Cloud cluster or attachment of an existing MKE cluster due to some project being stuck in the Terminating state.

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.14.0 including the Cluster releases 7.4.0, 6.20.0, and 5.21.0.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal
Hardware-related
vSphere

LCM
IAM
StackLight

Storage
Upgrade
Container Cloud web UI

Bare metal¶

[20745] Namespace deletion failure after managed cluster removal¶

Fixed in 2.16.0

Workaround:

Remove finalizers from all KaaSCephOperationRequest resources:

kubectl -n <managed-ns> get kaascephoperationrequest -o name | xargs -I % kubectl -n <managed-ns> patch % -p '{"metadata":{"finalizers":{}}}' --type=merge

Delete all KaaSCephOperationRequest resources:

kubectl -n <managed-ns> delete kaascephoperationrequest --all

[19786] Managed cluster deployment fails due to the dnsmasq-dhcpd logs overflow¶

Fixed in 2.15.0

A managed cluster deployment fails on long-running management clusters with BareMetalHost being stuck in the Preparing state and the ironic-conductor and ironic-api pods reporting the not enough disk space error due to the dnsmasq-dhcpd logs overflow.

Workaround:

Log in to the ironic-conductor pod.
Verify the free space in /volume/log/dnsmasq.
- If the free space on a volume is less than 10%:
  1. Manually delete log files in /volume/log/dnsmasq/.
  2. Scale down the dnsmasq pod to 0 replicas:
    kubectl -n kaas scale deployment dnsmasq --replicas=0
  3. Scale up the dnsmasq pod to 1 replica:
    kubectl -n kaas scale deployment dnsmasq --replicas=1
- If the volume has enough space, assess the Ironic logs to identify the root cause of the issue.

[17792] Full preflight fails with a timeout waiting for BareMetalHost¶

If you run bootstrap.sh preflight with KAAS_BM_FULL_PREFLIGHT=true, the script fails with the following message:

preflight check failed: preflight full check failed: \
error waiting for BareMetalHosts to power on: \
timed out waiting for the condition

Workaround:

Unset full preflight using the unset KAAS_BM_FULL_PREFLIGHT environment variable.
Rerun bootstrap.sh preflight that executes fast preflight instead.

vSphere¶

[19737] The vSphere VM template build hangs with an empty kickstart file¶

Fixed in 2.15.0

On the vSphere deployments with the RHEL 8.4 seed node, the VM template build for deployment hangs because of an empty kickstart file provided to the VM. In this case, the VMware web console displays the following error for the affected VM:

Kickstart file /run/install/ks.cfg is missing

The fix for the issue is implemented in the latest version of the Packer image for the VM template build.

Workaround:

Open bootstrap.sh in the kaas-bootstrap folder for editing.
Update the Docker image tag for the VSPHERE_PACKER_DOCKER_IMAGE variable to v1.0-39.
Save edits and restart the VM template build:
```
./bootstrap.sh vsphere_template
```

[19468] ‘Failed to remove finalizer from machine’ error during cluster deletion¶

Fixed in 2.15.0

If a RHEL license is removed before the related managed cluster is deleted, the cluster deletion hangs with the following Machine object error:

Failed to remove finalizer from machine ...
failed to get RHELLicense object

As a workaround, recreate the removed RHEL license object with the same name using the Container Cloud web UI or API.

Warning

The kubectl apply command automatically saves the applied data as plain text into the kubectl.kubernetes.io/last-applied-configuration annotation of the corresponding object. This may result in revealing sensitive data in this annotation when creating or modifying the object.

Therefore, do not use kubectl apply on this object. Use kubectl create, kubectl patch, or kubectl edit instead.

If you used kubectl apply on this object, you can remove the kubectl.kubernetes.io/last-applied-configuration annotation from the object using kubectl edit.

[14080] Node leaves the cluster after IP address change¶

Note

Moving forward, the workaround for this issue will be moved from Release Notes to Operations Guide: Troubleshooting.

A vSphere-based management cluster bootstrap fails due to a node leaving the cluster after an accidental IP address change.

The issue may affect a vSphere-based cluster only when IPAM is not enabled and IP addresses assignment to the vSphere virtual machines is done by a DHCP server present in the vSphere network.

By default, a DHCP server keeps lease of the IP address for 30 minutes. Usually, a VM dhclient prolongs such lease by frequent DHCP requests to the server before the lease period ends. The DHCP prolongation request period is always less than the default lease time on the DHCP server, so prolongation usually works. But in case of network issues, for example, when dhclient from the VM cannot reach the DHCP server, or the VM is being slowly powered on for more than the lease time, such VM may lose its assigned IP address. As a result, it obtains a new IP address.

Container Cloud does not support network reconfiguration after the IP of the VM has been changed. Therefore, such issue may lead to a VM leaving the cluster.

Symptoms:

One of the nodes is in the NodeNotReady or down state:
```
kubectl get nodes -o wide
docker node ls
```

The UCP Swarm manager logs on the healthy manager node contain the following example error:

docker logs -f ucp-swarm-manager

level=debug msg="Engine refresh failed" id="<docker node ID>|<node IP>: 12376"

If the affected node is manager:

The output of the docker info command contains the following example error:

Error: rpc error: code = Unknown desc = The swarm does not have a leader. \
It's possible that too few managers are online. \
Make sure more than half of the managers are online.

The UCP controller logs contain the following example error:

docker logs -f ucp-controller

"warning","msg":"Node State Active check error: \
Swarm Mode Manager health check error: \
info: Cannot connect to the Docker daemon at tcp://<node IP>:12376. \
Is the docker daemon running?

On the affected node, the IP address on the first interface eth0 does not match the IP address configured in Docker. Verify the Node Address field in the output of the docker info command.
The following lines are present in /var/log/messages:
```
dhclient[<pid>]: bound to <node IP> -- renewal in 1530 seconds
```
If there are several lines where the IP is different, the node is affected.

Workaround:

Select from the following options:

Bind IP addresses for all machines to their MAC addresses on the DHCP server for the dedicated vSphere network. In this case, VMs receive only specified IP addresses that never change.
Remove the Container Cloud node IPs from the IP range on the DHCP server for the dedicated vSphere network and configure the first interface eth0 on VMs with a static IP address.
If a managed cluster is affected, redeploy it with IPAM enabled for new machines to be created and IPs to be assigned properly.

LCM¶

[6066] Helm releases get stuck in FAILED or UNKNOWN state¶

Note

The issue affects only Helm v2 releases and is addressed for Helm v3. Starting from Container Cloud 2.19.0, all Helm releases are switched to v3.

During a management, regional, or managed cluster deployment, Helm releases may get stuck in the FAILED or UNKNOWN state although the corresponding machines statuses are Ready in the Container Cloud web UI. For example, if the StackLight Helm release fails, the links to its endpoints are grayed out in the web UI. In the cluster status, providerStatus.helm.ready and providerStatus.helm.releaseStatuses.<releaseName>.success are false.

HelmBundle cannot recover from such states and requires manual actions. The workaround below describes the recovery steps for the stacklight release that got stuck during a cluster deployment. Use this procedure as an example for other Helm releases as required.

Workaround:

Verify the failed release has the UNKNOWN or FAILED status in the HelmBundle object:

kubectl --kubeconfig <regionalClusterKubeconfigPath> get helmbundle <clusterName> -n <clusterProjectName> -o=jsonpath={.status.releaseStatuses.stacklight}

In the command above and in the steps below, replace the parameters
enclosed in angle brackets with the corresponding values of your cluster.

Example of system response:

stacklight:
attempt: 2
chart: ""
finishedAt: "2021-02-05T09:41:05Z"
hash: e314df5061bd238ac5f060effdb55e5b47948a99460c02c2211ba7cb9aadd623
message: '[{"occurrence":1,"lastOccurrenceDate":"2021-02-05 09:41:05","content":"error
  updating the release: rpc error: code = Unknown desc = customresourcedefinitions.apiextensions.k8s.io
  \"helmbundles.lcm.mirantis.com\" already exists"}]'
notes: ""
status: UNKNOWN
success: false
version: 0.1.2-mcp-398

kubectl --kubeconfig <affectedClusterKubeconfigPath> exec -n kube-system -it helm-controller-0 sh -c tiller

Download the Helm v3 binary. For details, see official Helm documentation.
Remove the failed release:
```
helm delete <failed-release-name>
```
For example:
```
helm delete stacklight
```
Once done, the release triggers for redeployment.

IAM¶

[21024] Adding a custom certificate for Keycloak hangs with a timeout warning¶

Fixed in 2.15.0

Adding a custom certificate for Keycloak using the container-cloud binary hangs with the failed to wait for OIDC certificate to be updated timeout warning. The readiness check fails due to a wrong condition.

Ignore the timeout warning. If you can log in to the Container Cloud web UI, the certificate has been applied successfully.

[18331] Keycloak admin console menu disappears on ‘Add identity provider’ page¶

Fixed in 2.18.0

During configuration of an identity provider SAML using the Add identity provider menu of the Keycloak admin console, the page style breaks as well as the Save and Cancel buttons disappear.

Workaround:

Log in to the Keycloak admin console.
In the sidebar menu, switch to the Master realm.
Navigate to Realm Settings > Themes.
In the Admin Console Theme drop-down menu, select keycloak.
Click Save and refresh the browser window to apply the changes.

StackLight¶

[18933] Alerta pods fail to pass the readiness check¶

Fixed in 2.15.0

Occasionally, an Alerta pod may be not Ready even if Patroni, the Alerta backend, operates correctly. In this case, some of the following errors may appear in the Alerta logs:

2021-10-25 13:10:55,865 DEBG 'nginx' stdout output:
2021/10/25 13:10:55 [crit] 25#25: *17408 connect() to unix:/tmp/uwsgi.sock failed (2: No such file or directory) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /api/config HTTP/1.1", upstream: "uwsgi://unix:/tmp/uwsgi.sock:", host: "127.0.0.1:8080"
ip=\- [\25/Oct/2021:13:10:55 +0000] "\GET /api/config HTTP/1.1" \502 \157 "\-" "\python-requests/2.24.0"
/web | /api/config | > GET /api/config HTTP/1.1

2021-11-11 00:02:23,969 DEBG 'nginx' stdout output:
2021/11/11 00:02:23 [error] 23#23: *2014 connect() to unix:/tmp/uwsgi.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 172.16.37.243, server: , request: "GET /api/services HTTP/1.1", upstream: "uwsgi://unix:/tmp/uwsgi.sock:", host: "10.233.113.143:8080"
ip=\- [\11/Nov/2021:00:02:23 +0000] "\GET /api/services HTTP/1.1" \502 \157 "\-" "\kube-probe/1.20+"
/web | /api/services | > GET /api/services HTTP/1.1

As a workaround, manually restart the affected Alerta pods:

kubectl delete pod -n stacklight <POD_NAME>

[19682] URLs in Salesforce alerts use HTTP for IAM with enabled TLS¶

Fixed in 2.15.0

Prometheus web UI URLs in StackLight notifications sent to Salesforce use a wrong protocol: HTTP instead of HTTPS. The issue affects deployments with TLS enabled for IAM.

The workaround is to manually change the URL protocol in the web browser.

Storage¶

[20312] Creation of ceph-based PVs gets stuck in Pending state¶

The csi-rbdplugin-provisioner pod (csi-provisioner container) may show constant retries attempting to create a PV if the csi-rbdplugin-provisioner pod was scheduled and started on a node with no connectivity to the Ceph storage. As a result, creation of a Ceph-based persistent volume (PV) may get stuck in the Pending state.

As a workaround manually specify the affinity or toleration rules for the csi-rbdplugin-provisioner pod.

Workaround:

On the managed cluster, open the rook-ceph-operator-config map for editing:
```
kubectl edit configmap -n rook-ceph rook-ceph-operator-config
```
To avoid spawning pods on the nodes where this is not needed, set the provisioner node affinity specifying the required node labels. For example:
```
CSI_PROVISIONER_NODE_AFFINITY: "role=storage-node; storage=rook, ceph"
```

Note

If needed, you can also specify CSI_PROVISIONER_TOLERATIONS tolerations. For example:

CSI_PROVISIONER_TOLERATIONS: |
  - effect: NoSchedule
    key: node-role.kubernetes.io/controlplane
    operator: Exists
  - effect: NoExecute
    key: node-role.kubernetes.io/etcd
    operator: Exists

[20355] KaaSCephOperationRequest is cached after recreation with the same name¶

Fixed in 2.15.0

When creating a new KaaSCephOperationRequest CR with the same name specified in metadata.name as in the previous KaaSCephOperationRequest CR, even if the previous request was deleted manually, the new request includes information about the previous actions and is in the Completed phase. In this case, no removal is performed.

Workaround:

On the management cluster, manually delete the old KaasCephOperationRequest CR with the same metadata.name:
```
kubectl -n ceph-lcm-mirantis delete KaasCephOperationRequest <name>
```
On the managed cluster, manually delete the old CephOsdRemoveRequest with the same metadata.name:
```
kubectl -n ceph-lcm-mirantis delete CephOsdRemoveRequest <name>
```

[20298] Spec validation failing during KaaSCephOperationRequest creation¶

Fixed in 2.15.0

Spec validation may fail with the following error when creating a KaaSCephOperationRequest CR:

The KaaSCephOperationRequest "test-remove-osd" is invalid: spec: Invalid value: 1:
spec in body should have at most 1 properties

Workaround:

On the management cluster, open the kaascephoperationrequests.kaas.mirantis.com CRD for editing:
```
kubectl edit crd kaascephoperationrequests.kaas.mirantis.com
```
Remove maxProperties: 1 and minProperties: 1 from spec.versions[0].schema.openAPIV3Schema.properties.spec:
```
spec:
  maxProperties: 1
  minProperties: 1
```

[19645] Ceph OSD removal request failure during ‘Processing’¶

Fixed in 2.15.0

Ocassionally, when Processing a Ceph OSD removal request, KaaSCephOperationRequest retries the osd stop command without an interval, which leads to removal request failure.

As a workaround create a new request to proceed with the Ceph OSD removal.

[19574] Ceph OSD removal does not clean up device used for multiple OSDs¶

Fixed in 2.15.0

When executing a Ceph OSD removal request to remove Ceph OSDs placed on one disk, the request completes without errors but the device itself still keeps the old LVM partitions. As a result, Rook cannot use such device.

The workaround is to manually clean up the affected device as described in Rook documentation: Zapping Devices.

Upgrade¶

[20459] Cluster upgrade fails with the certificate error during Ansible update¶

Fixed in 2.15.0

An upgrade of a management or regional cluster originally deployed using the Container Cloud release earlier than 2.8.0 fails with error setting certificate verify locations during Ansible update if a machine contains /usr/local/share/ca-certificates/mcc.crt, which is either empty or invalid. Managed clusters are not affected.

Workaround:

On every machine of the affected management or regional cluster:

Delete /usr/local/share/ca-certificates/mcc.crt.

In /etc/lcm/environment, remove the following line:

export SSL_CERT_FILE="/usr/local/share/ca-certificates/mcc.crt"

Restart lcm-agent:

systemctl restart lcm-agent-v0.3.0-104-gb7f5e8d8

[20455] Cluster upgrade fails on the LCMMachine CRD update¶

An upgrade of a management or regional cluster originally deployed using the Container Cloud release earlier than 2.8.0 fails with:

The LCM Agent version not updating from v0.3.0-67-g25ab9f1a to v0.3.0-105-g6fb89599

The following error message appearing in the events of the related LCMMachine:

kubectl describe lcmmachine <machineName>

Failed to upgrade agent: failed to update agent upgrade status: \
LCMMachine.lcm.mirantis.com "master-0" is invalid: \
status.lcmAgentUpgradeStatus.finishedAt: Invalid value: "null": \
status.lcmAgentUpgradeStatus.finishedAt in body must be of type string: "null"

As a workaround, change the preserveUnknownFields value for the LCMMachine CRD to false:

kubectl patch crd lcmmachines.lcm.mirantis.com -p '{"spec":{"preserveUnknownFields":false}}'

[4288] Equinix and MOS managed clusters update failure¶

Note

Moving forward, the workaround for this issue will be moved from Release Notes to Operations Guide: Troubleshooting.

The Equinix Metal and MOS-based managed clusters may fail to update to the latest Cluster release with kubelet being stuck and reporting authorization errors.

The cluster is affected by the issue if you see the Failed to make webhook authorizer request: context canceled error in the kubelet logs:

docker logs ucp-kubelet --since 5m 2>&1 | grep 'Failed to make webhook authorizer request: context canceled'

As a workaround, restart the ucp-kubelet container on the affected node(s):

ctr -n com.docker.ucp snapshot rm ucp-kubelet
docker rm -f ucp-kubelet

Note

Ignore failures in the output of the first command, if any.

[16379,23865] Cluster update fails with the FailedMount warning¶

Fixed in 2.19.0

An Equinix-based management or managed cluster fails to update with the FailedAttachVolume and FailedMount warnings.

Workaround:

Verify that the description of the pods that failed to run contain the FailedMount events:
```
kubectl -n <affectedProjectName> describe pod <affectedPodName>
```
- <affectedProjectName> is the Container Cloud project name where the pods failed to run
- <affectedPodName> is a pod name that failed to run in this project
In the pod description, identify the node name where the pod failed to run.
Verify that the csi-rbdplugin logs of the affected node contain the rbd volume mount failed: <csi-vol-uuid> is being used error. The <csi-vol-uuid> is a unique RBD volume name.
1. Identify csiPodName of the corresponding csi-rbdplugin:
```
kubectl -n rook-ceph get pod -l app=csi-rbdplugin \
-o jsonpath='{.items[?(@.spec.nodeName == "<nodeName>")].metadata.name}'
```
2. Output the affected csiPodName logs:
```
kubectl -n rook-ceph logs <csiPodName> -c csi-rbdplugin
```
Scale down the affected StatefulSet or Deployment of the pod that fails to init to 0 replicas.

On every csi-rbdplugin pod, search for stuck csi-vol:

for pod in `kubectl -n rook-ceph get pods|grep rbdplugin|grep -v provisioner|awk '{print $1}'`; do
  echo $pod
  kubectl exec -it -n rook-ceph $pod -c csi-rbdplugin -- rbd device list | grep <csi-vol-uuid>
done

Unmap the affected csi-vol:
```
rbd unmap -o force /dev/rbd
```
The /dev/rbd value is a mapped RBD volume that uses csi-vol.

Delete volumeattachment of the affected pod:

kubectl get volumeattachments | grep <csi-vol-uuid>
kubectl delete volumeattacmhent <id>

Scale up the affected StatefulSet or Deployment back to the original number of replicas and wait until its state is Running.

Container Cloud web UI¶

[249] A newly created project does not display in the Container Cloud web UI¶

Affects only Container Cloud 2.18.0 and earlier

Components versions¶

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.14.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

**Container Cloud release components versions**¶
Component	Application/Service	Version
AWS ^Updated	aws-provider	1.27.6
	aws-credentials-controller	1.27.6
Azure ^Updated	azure-provider	1.27.6
	azure-credentials-controller	1.27.6
Bare metal	ambassador ^Updated	1.20.1-alpine
	baremetal-operator ^Updated	5.2.7
	baremetal-public-api ^Updated	5.2.7
	baremetal-provider ^Updated	1.27.6
	ironic ^Updated	victoria-bionic-20211103083724
	ironic-operator	base-bionic-20210930105000
	kaas-ipam ^Updated	base-bionic-20211028140230
	local-volume-provisioner ^Updated	2.5.0-mcp
	mariadb	10.4.17-bionic-20210617085111
IAM	iam ^Updated	2.4.10
	iam-controller ^Updated	1.27.6
	keycloak	12.0.0
Container Cloud	admission-controller ^Updated	1.27.6
	agent-controller ^Updated	1.27.6
	byo-credentials-controller ^Updated	1.27.6
	byo-provider ^Updated	1.27.6
	kaas-public-api ^Updated	1.27.6
	kaas-exporter ^Updated	1.27.6
	kaas-ui ^Updated	1.27.8
	lcm-controller ^Updated	0.3.0-105-g6fb89599
	mcc-cache ^Updated	1.27.6
	portforward-controller ^Updated	1.27.6
	proxy-controller ^Updated	1.27.6
	rbac-controller ^Updated	1.27.6
	release-controller ^Updated	1.27.6
	rhellicense-controller ^Updated	1.27.6
	squid-proxy	0.0.1-5
	user-controller ^New	1.27.9
Equinix Metal ^Updated	equinix-provider	1.27.6
	equinix-credentials-controller	1.27.6
OpenStack ^Updated	openstack-provider	1.27.6
	os-credentials-controller	1.27.6
VMware vSphere ^Updated	vsphere-provider	1.27.6
	vsphere-credentials-controller	1.27.6

Artifacts¶

This section lists the components artifacts of the Mirantis Container Cloud release 2.14.0.

Bare metal artifacts
Core artifacts
IAM artifacts

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible ^Updated.

Bare metal artifacts¶
Artifact	Component	Path
Binaries	baremetal-operator ^Updated	https://binary.mirantis.com/bm/helm/baremetal-operator-5.2.7.tgz
	baremetal-public-api ^Updated	https://binary.mirantis.com/bm/helm/baremetal-public-api-5.2.7.tgz
	ironic-python-agent-bionic.kernel	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-victoria-bionic-debug-20210817124316
	ironic-python-agent-bionic.initramfs	https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-victoria-bionic-debug-20210817124316
	kaas-ipam ^Updated	https://binary.mirantis.com/bm/helm/kaas-ipam-5.2.7.tgz
	local-volume-provisioner ^Updated	https://binary.mirantis.com/bm/helm/local-volume-provisioner-2.5.0-mcp.tgz
	provisioning_ansible	https://binary.mirantis.com/bm/bin/ansible/provisioning_ansible-0.1.1-82-342bd22.tgz
	target ubuntu system	https://binary.mirantis.com/bm/bin/efi/ubuntu/tgz-bionic-20210622161844
Docker images	ambassador ^Updated	mirantis.azurecr.io/lcm/nginx:1.20.1-alpine
	baremetal-operator	mirantis.azurecr.io/bm/baremetal-operator:base-bionic-20211005112459
	dnsmasq	mirantis.azurecr.io/general/dnsmasq:focal-20210617094827
	ironic ^Updated	mirantis.azurecr.io/openstack/ironic:victoria-bionic-20211103083724
	ironic-inspector ^Updated	mirantis.azurecr.io/openstack/ironic-inspector:victoria-bionic-20211103083724
	ironic-operator	mirantis.azurecr.io/bm/ironic-operator:base-bionic-20210930105000
	ironic-prometheus-exporter	mirantis.azurecr.io/stacklight/ironic-prometheus-exporter:0.1-20210608113804
	kaas-ipam ^Updated	mirantis.azurecr.io/bm/kaas-ipam:base-bionic-20211028140230
	mariadb	mirantis.azurecr.io/general/mariadb:10.4.17-bionic-20210617085111
	syslog-ng	mirantis.azurecr.io/bm/syslog-ng:base-bionic-20210617094817

Core artifacts¶
Artifact	Component	Paths
Bootstrap tarball ^Updated	bootstrap-linux	https://binary.mirantis.com/core/bin/bootstrap-linux-1.27.6.tar.gz
	bootstrap-darwin	https://binary.mirantis.com/core/bin/bootstrap-darwin-1.27.6.tar.gz
Helm charts ^Updated	admission-controller	https://binary.mirantis.com/core/helm/admission-controller-1.27.6.tgz
	agent-controller	https://binary.mirantis.com/core/helm/agent-controller-1.27.6.tgz
	aws-credentials-controller	https://binary.mirantis.com/core/helm/aws-credentials-controller-1.27.6.tgz
	aws-provider	https://binary.mirantis.com/core/helm/aws-provider-1.27.6.tgz
	azure-credentials-controller	https://binary.mirantis.com/core/helm/azure-credentials-controller-1.27.6.tgz
	azure-provider	https://binary.mirantis.com/core/helm/azure-provider-1.27.6.tgz
	baremetal-provider	https://binary.mirantis.com/core/helm/baremetal-provider-1.27.6.tgz
	byo-credentials-controller	https://binary.mirantis.com/core/helm/byo-credentials-controller-1.27.6.tgz
	byo-provider	https://binary.mirantis.com/core/helm/byo-provider-1.27.6.tgz
	equinix-credentials-controller	https://binary.mirantis.com/core/helm/equinix-credentials-controller-1.27.6.tgz
	equinix-provider	https://binary.mirantis.com/core/helm/equinix-provider-1.27.6.tgz
	equinixmetalv2-provider ^New	https://binary.mirantis.com/core/helm/equinixmetalv2-provider-1.27.6.tgz
	iam-controller	https://binary.mirantis.com/core/helm/iam-controller-1.27.6.tgz
	kaas-exporter	https://binary.mirantis.com/core/helm/kaas-exporter-1.27.6.tgz
	kaas-public-api	https://binary.mirantis.com/core/helm/kaas-public-api-1.27.6.tgz
	kaas-ui	https://binary.mirantis.com/core/helm/kaas-ui-1.27.8.tgz
	lcm-controller	https://binary.mirantis.com/core/helm/lcm-controller-1.27.6.tgz
	mcc-cache	https://binary.mirantis.com/core/helm/mcc-cache-1.27.6.tgz
	openstack-provider	https://binary.mirantis.com/core/helm/openstack-provider-1.27.6.tgz
	os-credentials-controller	https://binary.mirantis.com/core/helm/os-credentials-controller-1.27.6.tgz
	portforward-controller	https://binary.mirantis.com/core/helm/portforward-controller-1.27.6.tgz
	proxy-controller	https://binary.mirantis.com/core/helm/proxy-controller-1.27.6.tgz
	rbac-controller	https://binary.mirantis.com/core/helm/rbac-controller-1.27.6.tgz
	release-controller	https://binary.mirantis.com/core/helm/release-controller-1.27.6.tgz
	rhellicense-controller	https://binary.mirantis.com/core/helm/rhellicense-controller-1.27.6.tgz
	squid-proxy	https://binary.mirantis.com/core/helm/squid-proxy-1.27.6.tgz
	vsphere-credentials-controller	https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.27.6.tgz
	vsphere-provider	https://binary.mirantis.com/core/helm/vsphere-provider-1.27.6.tgz
	user-controller ^New	https://binary.mirantis.com/core/helm/user-controller-1.27.9.tgz
Docker images	admission-controller ^Updated	mirantis.azurecr.io/core/admission-controller:1.27.6
	agent-controller ^Updated	mirantis.azurecr.io/core/agent-controller:1.27.6
	aws-cluster-api-controller ^Updated	mirantis.azurecr.io/core/aws-cluster-api-controller:1.27.6
	aws-credentials-controller ^Updated	mirantis.azurecr.io/core/aws-credentials-controller:1.27.6
	azure-cluster-api-controller ^Updated	mirantis.azurecr.io/core/azure-cluster-api-controller:1.27.6
	azure-credentials-controller ^Updated	mirantis.azurecr.io/core/azure-credentials-controller:1.27.6
	byo-cluster-api-controller ^Updated	mirantis.azurecr.io/core/byo-cluster-api-controller:1.27.6
	byo-credentials-controller ^Updated	mirantis.azurecr.io/core/byo-credentials-controller:1.27.6
	cluster-api-provider-baremetal ^Updated	mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.27.6
	cluster-api-provider-equinix ^Updated	mirantis.azurecr.io/core/cluster-api-provider-equinix:1.27.6
	equinix-credentials-controller ^Updated	mirantis.azurecr.io/core/equinix-credentials-controller:1.27.6
	frontend ^Updated	mirantis.azurecr.io/core/frontend:1.27.8
	iam-controller ^Updated	mirantis.azurecr.io/core/iam-controller:1.27.6
	kproxy ^Updated	mirantis.azurecr.io/lcm/kproxy:1.27.6
	lcm-controller ^Updated	mirantis.azurecr.io/core/lcm-controller:v0.3.0-105-g6fb89599
	nginx	mirantis.azurecr.io/lcm/nginx:1.20.1-alpine
	openstack-cluster-api-controller ^Updated	mirantis.azurecr.io/core/openstack-cluster-api-controller:1.27.6
	os-credentials-controller ^Updated	mirantis.azurecr.io/core/os-credentials-controller:1.27.6
	portforward-controller ^Updated	mirantis.azurecr.io/core/portforward-controller:1.27.6
	rbac-controller ^Updated	mirantis.azurecr.io/core/rbac-controller:1.27.6
	registry	mirantis.azurecr.io/lcm/registry:2.7.1
	release-controller ^Updated	mirantis.azurecr.io/core/release-controller:1.27.6
	rhellicense-controller ^Updated	mirantis.azurecr.io/core/rhellicense-controller:1.27.6
	squid-proxy	mirantis.azurecr.io/core/squid-proxy:0.0.1-5
	vsphere-cluster-api-controller ^Updated	mirantis.azurecr.io/core/vsphere-api-controller:1.27.6
	vsphere-credentials-controller ^Updated	mirantis.azurecr.io/core/vsphere-credentials-controller:1.27.6
	user-controller ^New	mirantis.azurecr.io/core/user-controller:1.27.9

IAM artifacts¶
Artifact	Component	Path
Binaries	hash-generate-linux	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux
	hash-generate-darwin	http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin
	iamctl-linux ^Updated	http://binary.mirantis.com/iam/bin/iamctl-0.5.4-linux
	iamctl-darwin ^Updated	http://binary.mirantis.com/iam/bin/iamctl-0.5.4-darwin
	iamctl-windows ^Updated	http://binary.mirantis.com/iam/bin/iamctl-0.5.4-windows
Helm charts	iam ^Updated	http://binary.mirantis.com/iam/helm/iam-2.4.10.tgz
	iam-proxy	http://binary.mirantis.com/iam/helm/iam-proxy-0.2.9.tgz
	keycloak_proxy	http://binary.mirantis.com/core/helm/keycloak_proxy-1.26.6.tgz
Docker images	api ^Updated	mirantis.azurecr.io/iam/api:0.5.4
	auxiliary ^Updated	mirantis.azurecr.io/iam/auxiliary:0.5.4
	kubernetes-entrypoint	mirantis.azurecr.io/iam/external/kubernetes-entrypoint:v0.3.1
	mariadb	mirantis.azurecr.io/general/mariadb:10.4.16-bionic-20201105025052
	keycloak ^Updated	mirantis.azurecr.io/iam/keycloak:0.5.4
	keycloak-gatekeeper	mirantis.azurecr.io/iam/keycloak-gatekeeper:7.1.3-2

See also

2.13.1¶

The Mirantis Container Cloud GA release 2.13.1 is based on 2.13.0 and:

Introduces support for the Cluster release 6.20.0 that is based on the Cluster release 5.20.0 and represents Mirantis OpenStack for Kubernetes (MOS) 21.6. This Cluster release is based on Mirantis Kubernetes Engine 3.3.12 with Kubernetes 1.18 and Mirantis Container Runtime 20.10.6.
Supports the latest Cluster releases 7.2.0 and 5.20.0.
Supports deprecated Cluster releases 7.2.0, 6.19.0, and 5.19.0 that will become unsupported in the following Container Cloud releases.

Caution

Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

For details about the Container Cloud release 2.13.1, refer to its parent release 2.13.0.

See also

2.13.0¶

The Mirantis Container Cloud GA release 2.13.0:

Introduces support for the Cluster release 7.3.0 that is based on Mirantis Container Runtime 20.10.6 and Mirantis Kubernetes Engine 3.4.5 with Kubernetes 1.20.
Introduces support for the Cluster release 5.20.0 that is based on Mirantis Kubernetes Engine 3.3.12 with Kubernetes 1.18 and Mirantis Container Runtime 20.10.6.
Supports the Cluster release 6.19.0 that is based on the Cluster release 5.19.0 and represents Mirantis OpenStack for Kubernetes (MOS) 21.5.
Supports deprecated Cluster releases 5.19.0, 6.18.0, and 7.2.0 that will become unsupported in the following Container Cloud releases.

Caution

Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.13.0.

Enhancements¶

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.13.0. For the list of enhancements in the Cluster releases 7.3.0 and 5.20.0 that are supported by the Container Cloud release 2.13.0, see the Cluster releases (managed).

Configuration of multiple DHCP ranges for bare metal clusters
Updated RAM requirements for management and regional clusters

Configuration of multiple DHCP ranges for bare metal clusters¶

Implemented the possibility to configure multiple DHCP ranges using the bare metal Subnet resources to facilitate multi-rack and other types of distributed bare metal datacenter topologies. The dnsmasq DHCP server used for host provisioning in Container Cloud now supports working with multiple L2 segments through DHCP relay capable network routers.

To configure DHCP ranges for dnsmasq, create the Subnet objects tagged with the ipam/SVC-dhcp-range label while setting up subnets for a managed cluster using Container Cloud CLI.

Learn more

Configure multiple DHCP address ranges

Updated RAM requirements for management and regional clusters¶

To improve the Container Cloud performance and stability, increased RAM requirements for management and regional clusters from 16 to 24 GB for all supported cloud providers except bare metal, with the corresponding flavor changes for the AWS and Azure providers:

AWS: updated the instance type from c5d.2xlarge to c5d.4xlarge
Azure: updated the VM size from Standard_F8s_v2 to Standard_F16s_v2

For the Container Cloud managed clusters, requirements remain the same.

Addressed issues¶

The following issues have been addressed in the Mirantis Container Cloud release 2.13.0 along with the Cluster releases 7.3.0 and 5.20.0.

[17705][Azure] Fixed the issue with the failure to deploy more than 62 Azure worker nodes.
[17938][bare metal] Fixed the issue with the bare metal host profile being stuck in the match profile state during bootstrap.
[17960][bare metal] Fixed the issue with overflow of the Ironic storage volume causing a StackLight alert being triggered for the ironic-aio-pvc volume filling up.
[17981][bare metal] Fixed the issue with failure to redeploy a bare metal node with an mdadm-based raid1 enabled due to insufficient cleanup of RAID devices.
[17359][regional cluster] Fixed the issue with failure to delete an AWS-based regional cluster due to the issue with the cluster credential deletion.
[18193][upgrade] Fixed the issue with failure to upgrade an Equinix Metal or baremetal-based management cluster with Ceph cluster being not ready.
[18076][upgrade] Fixed the issue with StackLight update failure on managed cluster with logging disabled after changing NodeSelector.
[17771][StackLight] Fixed the issue with the Watchdog alert not routing to Salesforce by default.

If you have applied the workaround as described in StackLight known issues: 17771, revert it after updating the Cluster releases to 5.20.0, 6.20.0, or 7.3.0:
1. Open the StackLight configuration manifest as described in StackLight configuration procedure.
2. In alertmanagerSimpleConfig.salesForce:
  - remove the match and march_re parameters since they are deprecated
  - remove the matchers parameter since it changes the default settings

Known issues¶

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.13.0 including the Cluster releases 7.3.0, 6.19.0, and 5.20.0.

For other issues that can occur while deploying and operating a Container Cloud cluster, see Deployment Guide: Troubleshooting and Operations Guide: Troubleshooting.

Note

This section also outlines still valid known issues from previous Container Cloud releases.

Bare metal
OpenStack
vSphere
LCM
IAM

StackLight
Storage
Upgrade
Container Cloud web UI

Bare metal¶

[18752] Bare metal hosts in ‘provisioned registration error’ state after update¶

Note

Moving forward, the workaround for this issue will be moved from Release Notes to Operations Guide: Troubleshooting.

After update of a management or managed cluster created using the Container Cloud release earlier than 2.6.0, a bare metal host state is Provisioned in the Container Cloud web UI while having the error state in logs with the following message:

status:
  errorCount: 1
  errorMessage: 'Host adoption failed: Error while attempting to adopt node  7a8d8aa7-e39d-48ec-98c1-ed05eacc354f:
    Validation of image href http://10.10.10.10/images/stub_image.qcow2 failed,
    reason: Got HTTP code 404 instead of 200 in response to HEAD request..'
  errorType: provisioned registration error

The issue is caused by the image URL pointing to an unavailable resource due to the URI IP change during update. As a workaround, update URLs for the bare metal host status and spec with the correct values that use a stable DNS record as a host.

Workaround:

Note

In the commands below, we update

Mirantis Container Cloud Documentation¶

Intended audience¶

Conventions¶

Technology Preview features¶

Documentation history¶

Product Overview¶

Reference Architecture¶

Overview¶

Container Cloud provider¶

Release Controller¶

Web UI¶

Bare metal¶

Bare metal components¶

Bare metal networking¶

IP Address Management¶

Management cluster networking¶

Managed cluster networking¶

Host networking¶

Storage¶

Overview¶

Limitations¶

Addressing storage devices¶

Extended hardware configuration¶

Automatic upgrade of a host operating system¶

Built-in load balancing¶

Kubernetes lifecycle management¶

LCM custom resources¶

LCM Controller¶

LCM Agent¶

Helm Controller¶

Identity and access management¶

External identity provider integration¶

Authentication and authorization¶

Implementation flow¶

Kubernetes CLI authentication flow¶

Monitoring¶

Deployment architecture¶

Authentication flow¶

Supported features¶

Monitored components¶

Storage-based log retention strategy¶

Outbound cluster metrics¶

StackLight proxy¶

Requirements¶

Reference hardware configuration¶

System requirements for the seed node¶

Network fabric¶

DHCP range requirements for PXE¶

Management cluster storage¶

Proxy and cache support¶

MKE API limitations¶

MKE configuration management¶

Deployment Guide¶

Deploy a Container Cloud management cluster¶

Introduction¶

Overview of the deployment workflow¶

Set up a bootstrap cluster¶

Deploy a management cluster using the Container Cloud API¶

Configure a bare metal deployment¶

Configure BIOS on a bare metal host¶

Customize the default bare metal host profile¶

Configure NIC bonding¶

Separate PXE and management networks¶

Configure multiple DHCP address ranges¶

Enable dynamic IP allocation¶

Set a custom external IP address for the DHCP service¶

Configure optional cluster settings¶

Post-deployment steps¶

Troubleshooting¶

Troubleshoot the bootstrap region creation¶

Troubleshoot machines creation¶

Troubleshoot deployment stages¶

Collect the bootstrap logs¶

Requirements for a MITM proxy¶

Create initial users after a management cluster bootstrap¶

Troubleshooting¶

Troubleshoot the bootstrap node configuration¶

DNS settings¶

Default network addresses¶

Configure external identity provider for IAM¶