Mirantis Container Cloud Documentation

The documentation is intended to help operators understand the core concepts of the product.

The information provided in this documentation set is being constantly improved and amended based on the feedback and kind requests from our software consumers. This documentation set outlines description of the features supported within three latest Container Cloud minor releases and their supported Cluster releases, with a corresponding note Available since <release-version>.

The following table lists the guides included in the documentation set you are reading:

Guides list

Guide

Purpose

Reference Architecture

Learn the fundamentals of Container Cloud reference architecture to plan your deployment.

Deployment Guide

Deploy Container Cloud of a preferred configuration using supported deployment profiles tailored to the demands of specific business cases.

Operations Guide

Deploy and operate the Container Cloud managed clusters.

Release Compatibility Matrix

Deployment compatibility of the Container Cloud components versions for each product release.

Release Notes

Learn about new features and bug fixes in the current Container Cloud version as well as in the Container Cloud minor releases.

QuickStart Guides

Easy and lightweight instructions to get started with Container Cloud.

Intended audience

This documentation assumes that the reader is familiar with network and cloud concepts and is intended for the following users:

  • Infrastructure Operator

    • Is member of the IT operations team

    • Has working knowledge of Linux, virtualization, Kubernetes API and CLI, and OpenStack to support the application development team

    • Accesses Mirantis Container Cloud and Kubernetes through a local machine or web UI

    • Provides verified artifacts through a central repository to the Tenant DevOps engineers

  • Tenant DevOps engineer

    • Is member of the application development team and reports to line-of-business (LOB)

    • Has working knowledge of Linux, virtualization, Kubernetes API and CLI to support application owners

    • Accesses Container Cloud and Kubernetes through a local machine or web UI

    • Consumes artifacts from a central repository approved by the Infrastructure Operator

Conventions

This documentation set uses the following conventions in the HTML format:

Documentation conventions

Convention

Description

boldface font

Inline CLI tools and commands, titles of the procedures and system response examples, table titles.

monospaced font

Files names and paths, Helm charts parameters and their values, names of packages, nodes names and labels, and so on.

italic font

Information that distinguishes some concept or term.

Links

External links and cross-references, footnotes.

Main menu > menu item

GUI elements that include any part of interactive user interface and menu navigation.

Superscript

Some extra, brief information. For example, if a feature is available from a specific release or if a feature is in the Technology Preview development stage.

Note

The Note block

Messages of a generic meaning that may be useful to the user.

Caution

The Caution block

Information that prevents a user from mistakes and undesirable consequences when following the procedures.

Warning

The Warning block

Messages that include details that can be easily missed, but should not be ignored by the user and are valuable before proceeding.

See also

The See also block

List of references that may be helpful for understanding of some related tools, concepts, and so on.

Learn more

The Learn more block

Used in the Release Notes to wrap a list of internal references to the reference architecture, deployment and operation procedures specific to a newly implemented product feature.

Technology Preview features

A Technology Preview feature provides early access to upcoming product innovations, allowing customers to experiment with the functionality and provide feedback.

Technology Preview features may be privately or publicly available but neither are intended for production use. While Mirantis will provide assistance with such features through official channels, normal Service Level Agreements do not apply.

As Mirantis considers making future iterations of Technology Preview features generally available, we will do our best to resolve any issues that customers experience when using these features.

During the development of a Technology Preview feature, additional components may become available to the public for evaluation. Mirantis cannot guarantee the stability of such features. As a result, if you are using Technology Preview features, you may not be able to seamlessly update to subsequent product releases, as well as upgrade or migrate to the functionality that has not been announced as full support yet.

Mirantis makes no guarantees that Technology Preview features will graduate to generally available features.

Documentation history

The documentation set refers to Mirantis Container Cloud GA as to the latest released GA version of the product. For details about the Container Cloud GA minor releases dates, refer to Container Cloud releases.

Product Overview

Mirantis Container Cloud enables you to ship code faster by enabling speed with choice, simplicity, and security. Through a single pane of glass you can deploy, manage, and observe Kubernetes clusters on private clouds or bare metal infrastructure. Container Cloud provides the ability to leverage the following on premises cloud infrastructure: OpenStack, VMware, and bare metal.

The list of the most common use cases includes:

Multi-cloud

Organizations are increasingly moving toward a multi-cloud strategy, with the goal of enabling the effective placement of workloads over multiple platform providers. Multi-cloud strategies can introduce a lot of complexity and management overhead. Mirantis Container Cloud enables you to effectively deploy and manage container clusters (Kubernetes and Swarm) across multiple cloud provider platforms.

Hybrid cloud

The challenges of consistently deploying, tracking, and managing hybrid workloads across multiple cloud platforms is compounded by not having a single point that provides information on all available resources. Mirantis Container Cloud enables hybrid cloud workload by providing a central point of management and visibility of all your cloud resources.

Kubernetes cluster lifecycle management

The consistent lifecycle management of a single Kubernetes cluster is a complex task on its own that is made infinitely more difficult when you have to manage multiple clusters across different platforms spread across the globe. Mirantis Container Cloud provides a single, centralized point from which you can perform full lifecycle management of your container clusters, including automated updates and upgrades. Container Cloud also supports attachment of existing Mirantis Kubernetes Engine clusters that are not originally deployed by Container Cloud.

Highly regulated industries

Regulated industries need a fine level of access control granularity, high security standards and extensive reporting capabilities to ensure that they can meet and exceed the security standards and requirements. Mirantis Container Cloud provides for a fine-grained Role Based Access Control (RBAC) mechanism and easy integration and federation to existing identity management systems (IDM).

Logging, monitoring, alerting

A complete operational visibility is required to identify and address issues in the shortest amount of time – before the problem becomes serious. Mirantis StackLight is the proactive monitoring, logging, and alerting solution designed for large-scale container and cloud observability with extensive collectors, dashboards, trend reporting and alerts.

Storage

Cloud environments require a unified pool of storage that can be scaled up by simply adding storage server nodes. Ceph is a unified, distributed storage system designed for excellent performance, reliability, and scalability. Deploy Ceph utilizing Rook to provide and manage a robust persistent storage that can be used by Kubernetes workloads on the baremetal-based clusters.

Security

Security is a core concern for all enterprises, especially with more of our systems being exposed to the Internet as a norm. Mirantis Container Cloud provides for a multi-layered security approach that includes effective identity management and role based authentication, secure out of the box defaults and extensive security scanning and monitoring during the development process.

5G and Edge

The introduction of 5G technologies and the support of Edge workloads requires an effective multi-tenant solution to manage the underlying container infrastructure. Mirantis Container Cloud provides for a full stack, secure, multi-cloud cluster management and Day-2 operations solution that supports both on premises bare metal and cloud.

Reference Architecture

Overview

Mirantis Container Cloud is a set of microservices that are deployed using Helm charts and run in a Kubernetes cluster. Container Cloud is based on the Kubernetes Cluster API community initiative.

The following diagram illustrates an overview of Container Cloud and the clusters it manages:

_images/cluster-overview.png

All artifacts used by Kubernetes and workloads are stored on the Container Cloud content delivery network (CDN):

  • mirror.mirantis.com (Debian packages including the Ubuntu mirrors)

  • binary.mirantis.com (Helm charts and binary artifacts)

  • mirantis.azurecr.io (Docker image registry)

All Container Cloud components are deployed in the Kubernetes clusters. All Container Cloud APIs are implemented using the Kubernetes Custom Resource Definition (CRD) that represents custom objects stored in Kubernetes and allows you to expand Kubernetes API.

The Container Cloud logic is implemented using controllers. A controller handles the changes in custom resources defined in the controller CRD. A custom resource consists of a spec that describes the desired state of a resource provided by a user. During every change, a controller reconciles the external state of a custom resource with the user parameters and stores this external state in the status subresource of its custom resource.

Container Cloud regions

Unsupported since 2.25.0

Caution

Regional clusters are unsupported since Container Cloud 2.25.0. Mirantis does not perform functional integration testing of the feature and intends to remove the related code in Container Cloud 2.26.0. If you still require this feature, contact Mirantis support for further information.

Container Cloud can have several regions. A region is a physical location, for example, a data center, that has access to one or several cloud provider back ends. A separate regional cluster manages a region that can include multiple providers. A region must have a two-way (full) network connectivity between a regional cluster and a cloud provider back end. For example, an OpenStack VM must have access to the related regional cluster. And this regional cluster must have access to the OpenStack floating IPs and load balancers.

The following diagram illustrates the structure of the Container Cloud regions:

_images/regions.png
Container Cloud cluster types

The types of the Container Cloud clusters include:

Bootstrap cluster v2 Recommended since 2.25.0
  • Contains the Bootstrap web UI for the OpenStack and vSphere providers. The Bootstrap web UI support for the bare metal provider will be added in one of the following Container Cloud releases.

  • Runs the bootstrap process on a seed node that can be reused after the management cluster deployment for other purposes. For the OpenStack or vSphere provider, it can be an operator desktop computer. For the bare metal provider, this is a data center node.

  • Requires access to one of the following provider back ends: bare metal, OpenStack, or vSphere.

  • Initially, the bootstrap cluster is created with the following minimal set of components: Bootstrap Controller, public API charts, and the Bootstrap web UI.

  • The user can interact with the bootstrap cluster through the Bootstrap web UI or API to create the configuration for a management cluster and start its deployment. More specifically, the user performs the following operations:

    1. Select the provider, add provider credentials.

    2. Add proxy and SSH keys.

    3. Configure the cluster and machines.

    4. Deploy a management cluster.

  • The user can monitor the deployment progress of the cluster and machines.

  • After a successful deployment, the user can download the kubeconfig artifact of the provisioned cluster.

Bootstrap cluster v1 Deprecated since 2.25.0
  • Runs the bootstrap process on a seed node that can be reused after the management cluster deployment for other purposes. For the OpenStack or vSphere provider, it can be an operator desktop computer. For the bare metal provider, this is a data center node.

  • Requires access to one of the following provider back ends: OpenStack, vSphere, or bare metal.

  • Contains a minimum set of services to deploy management and regional clusters.

  • Is destroyed completely after a successful bootstrap.

Management and regional clusters
  • Management cluster:

    • Runs all public APIs and services including the web UIs of Container Cloud.

    • Does not require access to any provider back end.

  • Regional cluster:

    • Is combined with management cluster by default.

    • Runs the provider-specific services and internal API including LCMMachine and LCMCluster. Also, it runs an LCM controller for orchestrating managed clusters and other controllers for handling different resources.

    • Requires two-way access to a provider back end. The provider connects to a back end to spawn managed cluster nodes, and the agent running on the nodes accesses the regional cluster to obtain the deployment information.

    • Requires access to a management cluster to obtain user parameters.

    • Supports multi-regional deployments. For example, you can deploy a vSphere-based management cluster with vSphere-based and OpenStack-based regional clusters.

    Caution

    Regional clusters are unsupported since Container Cloud 2.25.0. Mirantis does not perform functional integration testing of the feature and intends to remove the related code in Container Cloud 2.26.0. If you still require this feature, contact Mirantis support for further information.

Management and regional clusters comprise Container Cloud as product. For deployment details, see Deployment Guide and Deploy an additional regional cluster (optional) sections for the required cloud provider.

Managed cluster
  • A Mirantis Kubernetes Engine (MKE) cluster that an end user creates using the Container Cloud web UI.

  • Requires access to a regional cluster. Each node of a managed cluster runs an LCM Agent that connects to the LCM machine of the regional cluster to obtain the deployment details.

  • Since 2.25.2, an attached MKE cluster that is not created using Container Cloud for vSphere-based clusters. In such case, nodes of the attached cluster do not contain LCM Agent. For supported MKE versions that can be attached to Container Cloud, see Release Compatibility Matrix.

  • Baremetal-based managed clusters support the Mirantis OpenStack for Kubernetes (MOSK) product. For details, see MOSK documentation.

All types of the Container Cloud clusters except the bootstrap cluster are based on the MKE and Mirantis Container Runtime (MCR) architecture. For details, see MKE and MCR documentation.

The following diagram illustrates the distribution of services between each type of the Container Cloud clusters:

_images/cluster-types.png

Cloud provider

The Mirantis Container Cloud provider is the central component of Container Cloud that provisions a node of a management, regional, or managed cluster and runs the LCM Agent on this node. It runs in a management and regional clusters and requires connection to a provider back end.

The Container Cloud provider interacts with the following types of public API objects:

Public API object name

Description

Container Cloud release object

Contains the following information about clusters:

  • Version of the supported Cluster release for a management and regional clusters

  • List of supported Cluster releases for the managed clusters and supported upgrade path

  • Description of Helm charts that are installed on the management and regional clusters depending on the selected provider

Cluster release object

  • Provides a specific version of a management, regional, or managed cluster. Any Cluster release object, as well as a Container Cloud release object never changes, only new releases can be added. Any change leads to a new release of a cluster.

  • Contains references to all components and their versions that are used to deploy all cluster types:

    • LCM components:

      • LCM Agent

      • Ansible playbooks

      • Scripts

      • Description of steps to execute during a cluster deployment and upgrade

      • Helm Controller image references

    • Supported Helm charts description:

      • Helm chart name and version

      • Helm release name

      • Helm values

Cluster object

  • References the Credentials, KaaSRelease and ClusterRelease objects.

  • Is tied to a specific Container Cloud region and provider.

  • Represents all cluster-level resources. For example, for the OpenStack-based clusters, it represents networks, load balancer for the Kubernetes API, and so on. It uses data from the Credentials object to create these resources and data from the KaaSRelease and ClusterRelease objects to ensure that all lower-level cluster objects are created.

Machine object

  • References the Cluster object.

  • Represents one node of a managed cluster, for example, an OpenStack VM, and contains all data to provision it.

Credentials object

  • Contains all information necessary to connect to a provider back end.

  • Is tied to a specific Container Cloud region and provider.

PublicKey object

Is provided to every machine to obtain an SSH access.

The following diagram illustrates the Container Cloud provider data flow:

_images/provider-dataflow.png

The Container Cloud provider performs the following operations in Container Cloud:

  • Consumes the below types of data from a management and regional cluster:

    • Credentials to connect to a provider back end

    • Deployment instructions from the KaaSRelease and ClusterRelease objects

    • The cluster-level parameters from the Cluster objects

    • The machine-level parameters from the Machine objects

  • Prepares data for all Container Cloud components:

    • Creates the LCMCluster and LCMMachine custom resources for LCM Controller and LCM Agent. The LCMMachine custom resources are created empty to be later handled by the LCM Controller.

    • Creates the HelmBundle custom resources for the Helm Controller using data from the KaaSRelease and ClusterRelease objects.

    • Creates service accounts for these custom resources.

    • Creates a scope in Identity and access management (IAM) for a user access to a managed cluster.

  • Provisions nodes for a managed cluster using the cloud-init script that downloads and runs the LCM Agent.

  • Installs Helm Controller as a Helm v3 chart.

Release Controller

The Mirantis Container Cloud Release Controller is responsible for the following functionality:

  • Monitor and control the KaaSRelease and ClusterRelease objects present in a management cluster. If any release object is used in a cluster, the Release Controller prevents the deletion of such an object.

  • Sync the KaaSRelease and ClusterRelease objects published at https://binary.mirantis.com/releases/ with an existing management cluster.

  • Trigger the Container Cloud auto-upgrade procedure if a new KaaSRelease object is found:

    1. Search for the managed clusters with old Cluster releases that are not supported by a new Container Cloud release. If any are detected, abort the auto-upgrade and display a corresponding note about an old Cluster release in the Container Cloud web UI for the managed clusters. In this case, a user must update all managed clusters using the Container Cloud web UI. Once all managed clusters are upgraded to the Cluster releases supported by a new Container Cloud release, the Container Cloud auto-upgrade is retriggered by the Release Controller.

    2. Trigger the Container Cloud release upgrade of all Container Cloud components in a management cluster. The upgrade itself is processed by the Container Cloud provider.

    3. Trigger the Cluster release upgrade of a management cluster to the Cluster release version that is indicated in the upgraded Container Cloud release version. The LCMCluster components, such as MKE, are upgraded before the HelmBundle components, such as StackLight or Ceph.

    4. Verify the regional cluster(s) status. If the regional cluster is ready, trigger the Cluster release upgrade of the regional cluster.

      Once a management cluster is upgraded, an option to update a managed cluster becomes available in the Container Cloud web UI. During a managed cluster update, all cluster components including Kubernetes are automatically upgraded to newer versions if available. The LCMCluster components, such as MKE, are upgraded before the HelmBundle components, such as StackLight or Ceph.

The Operator can delay the Container Cloud automatic upgrade procedure for a limited amount of time or schedule upgrade to run at desired hours or weekdays. For details, see Schedule Mirantis Container Cloud upgrades.

Container Cloud remains operational during the management and regional clusters upgrade. Managed clusters are not affected during this upgrade. For the list of components that are updated during the Container Cloud upgrade, see the Components versions section of the corresponding Container Cloud release in Release Notes.

When Mirantis announces support of the newest versions of Mirantis Container Runtime (MCR) and Mirantis Kubernetes Engine (MKE), Container Cloud automatically upgrades these components as well. For the maintenance window best practices before upgrade of these components, see MKE Documentation.

See also

Patch releases

Web UI

The Mirantis Container Cloud web UI is mainly designed to create and update the managed clusters as well as add or remove machines to or from an existing managed cluster.

You can use the Container Cloud web UI to obtain the management cluster details including endpoints, release version, and so on. The management cluster update occurs automatically with a new release change log available through the Container Cloud web UI.

The Container Cloud web UI is a JavaScript application that is based on the React framework. The Container Cloud web UI is designed to work on a client side only. Therefore, it does not require a special back end. It interacts with the Kubernetes and Keycloak APIs directly. The Container Cloud web UI uses a Keycloak token to interact with Container Cloud API and download kubeconfig for the management and managed clusters.

The Container Cloud web UI uses NGINX that runs on a management cluster and handles the Container Cloud web UI static files. NGINX proxies the Kubernetes and Keycloak APIs for the Container Cloud web UI.

Bare metal

The bare metal service provides for the discovery, deployment, and management of bare metal hosts.

The bare metal management in Mirantis Container Cloud is implemented as a set of modular microservices. Each microservice implements a certain requirement or function within the bare metal management system.

Bare metal components

The bare metal management solution for Mirantis Container Cloud includes the following components:

Bare metal components

Component

Description

OpenStack Ironic

The back-end bare metal manager in a standalone mode with its auxiliary services that include httpd, dnsmasq, and mariadb.

OpenStack Ironic Inspector

Introspects and discovers the bare metal hosts inventory. Includes OpenStack Ironic Python Agent (IPA) that is used as a provision-time agent for managing bare metal hosts.

Ironic Operator

Monitors changes in the external IP addresses of httpd, ironic, and ironic-inspector and automatically reconciles the configuration for dnsmasq, ironic, baremetal-provider, and baremetal-operator.

Bare Metal Operator

Manages bare metal hosts through the Ironic API. The Container Cloud bare-metal operator implementation is based on the Metal³ project.

Bare metal resources manager

Ensures that the bare metal provisioning artifacts such as the distribution image of the operating system is available and up to date.

cluster-api-provider-baremetal

The plugin for the Kubernetes Cluster API integrated with Container Cloud. Container Cloud uses the Metal³ implementation of cluster-api-provider-baremetal for the Cluster API.

HAProxy

Load balancer for external access to the Kubernetes API endpoint.

LCM Agent

Used for physical and logical storage, physical and logical network, and control over the life cycle of a bare metal machine resources.

Ceph

Distributed shared storage is required by the Container Cloud services to create persistent volumes to store their data.

MetalLB

Load balancer for Kubernetes services on bare metal. 1

Keepalived

Monitoring service that ensures availability of the virtual IP for the external load balancer endpoint (HAProxy). 1

IPAM

IP address management services provide consistent IP address space to the machines in bare metal clusters. See details in IP Address Management.

1(1,2)

For details, see Built-in load balancing.

The diagram below summarizes the following components and resource kinds:

  • Metal³-based bare metal management in Container Cloud (white)

  • Internal APIs (yellow)

  • External dependency components (blue)

_images/bm-component-stack.png
Bare metal networking

This section provides an overview of the networking configuration and the IP address management in the Mirantis Container Cloud on bare metal.

IP Address Management

Mirantis Container Cloud on bare metal uses IP Address Management (IPAM) to keep track of the network addresses allocated to bare metal hosts. This is necessary to avoid IP address conflicts and expiration of address leases to machines through DHCP.

Note

Only IPv4 address family is currently supported by Container Cloud and IPAM. IPv6 is not supported and not used in Container Cloud.

IPAM is provided by the kaas-ipam controller. Its functions include:

  • Allocation of IP address ranges or subnets to newly created clusters using SubnetPool and Subnet resources.

  • Allocation IP addresses to machines and cluster services at the request of baremetal-provider using the IpamHost and IPaddr resources.

  • Creation and maintenance of host networking configuration on the bare metal hosts using the IpamHost resources.

The IPAM service can support different networking topologies and network hardware configurations on the bare metal hosts.

In the most basic network configuration, IPAM uses a single L3 network to assign addresses to all bare metal hosts, as defined in Managed cluster networking.

You can apply complex networking configurations to a bare metal host using the L2 templates. The L2 templates imply multihomed host networking and enable you to create a managed cluster where nodes use separate host networks for different types of traffic. Multihoming is required to ensure the security and performance of a managed cluster.

Caution

Modification of L2 templates in use is allowed with a mandatory validation step from the Infrastructure Operator to prevent accidental cluster failures due to unsafe changes. The list of risks posed by modifying L2 templates includes:

  • Services running on hosts cannot reconfigure automatically to switch to the new IP addresses and/or interfaces.

  • Connections between services are interrupted unexpectedly, which can cause data loss.

  • Incorrect configurations on hosts can lead to irrevocable loss of connectivity between services and unexpected cluster partition or disassembly.

For details, see Modify network configuration on an existing machine.

Management cluster networking

The main purpose of networking in a Container Cloud management or regional cluster is to provide access to the Container Cloud Management API that consists of the Kubernetes API of the Container Cloud management and regional clusters and the Container Cloud LCM API. This API allows end users to provision and configure managed clusters and machines. Also, this API is used by LCM Agents in managed clusters to obtain configuration and report status.

The following types of networks are supported for the management and regional clusters in Container Cloud:

  • PXE network

    Enables PXE boot of all bare metal machines in the Container Cloud region.

    • PXE subnet

      Provides IP addresses for DHCP and network boot of the bare metal hosts for initial inspection and operating system provisioning. This network may not have the default gateway or a router connected to it. The PXE subnet is defined by the Container Cloud Operator during bootstrap.

      Provides IP addresses for the bare metal management services of Container Cloud, such as bare metal provisioning service (Ironic). These addresses are allocated and served by MetalLB.

  • Management network

    Connects LCM Agents running on the hosts to the Container Cloud LCM API. Serves the external connections to the Container Cloud Management API. The network is also used for communication between kubelet and the Kubernetes API server inside a Kubernetes cluster. The MKE components use this network for communication inside a swarm cluster.

    • LCM subnet

      Provides IP addresses for the Kubernetes nodes in the management cluster. This network also provides a Virtual IP (VIP) address for the load balancer that enables external access to the Kubernetes API of a management cluster. This VIP is also the endpoint to access the Container Cloud Management API in the management cluster.

      Provides IP addresses for the externally accessible services of Container Cloud, such as Keycloak, web UI, StackLight. These addresses are allocated and served by MetalLB.

  • Kubernetes workloads network

    Technology Preview

    Serves the internal traffic between workloads on the management cluster.

    • Kubernetes workloads subnet

      Provides IP addresses that are assigned to nodes and used by Calico.

  • Out-of-Band (OOB) network

    Connects to Baseboard Management Controllers of the servers that host the management cluster. The OOB subnet must be accessible from the management network through IP routing. The OOB network is not managed by Container Cloud and is not represented in the IPAM API.

Managed cluster networking

A Kubernetes cluster networking is typically focused on connecting pods on different nodes. On bare metal, however, the cluster networking is more complex as it needs to facilitate many different types of traffic.

Kubernetes clusters managed by Mirantis Container Cloud have the following types of traffic:

  • PXE network

    Enables the PXE boot of all bare metal machines in Container Cloud. This network is not configured on the hosts in a managed cluster. It is used by the bare metal provider to provision additional hosts in managed clusters and is disabled on the hosts after provisioning is done.

  • Life-cycle management (LCM) network

    Connects LCM Agents running on the hosts to the Container Cloud LCM API. The LCM API is provided by the regional or management cluster. The LCM network is also used for communication between kubelet and the Kubernetes API server inside a Kubernetes cluster. The MKE components use this network for communication inside a swarm cluster.

    When using the BGP announcement of the IP address for the cluster API load balancer, which is available as Technology Preview since Container Cloud 2.24.4, no segment stretching is required between Kubernetes master nodes. Also, in this scenario, the load balancer IP address is not required to match the LCM subnet CIDR address.

    • LCM subnet(s)

      Provides IP addresses that are statically allocated by the IPAM service to bare metal hosts. This network must be connected to the Kubernetes API endpoint of the regional cluster through an IP router. LCM Agents running on managed clusters will connect to the regional cluster API through this router. LCM subnets may be different per managed cluster as long as this connection requirement is satisfied. The Virtual IP (VIP) address for load balancer that enables access to the Kubernetes API of the managed cluster must be allocated from the LCM subnet.

    • Cluster API subnet

      Technology Preview

      Provides a load balancer IP address for external access to the cluster API. Mirantis recommends that this subnet stays unique per managed cluster.

  • Kubernetes workloads network

    Serves as an underlay network for traffic between pods in the managed cluster. Do not share this network between clusters.

    • Kubernetes workloads subnet(s)

      Provides IP addresses that are statically allocated by the IPAM service to all nodes and that are used by Calico for cross-node communication inside a cluster. By default, VXLAN overlay is used for Calico cross-node communication.

  • Kubernetes external network

    Serves ingress traffic to the managed cluster from the outside world. You can share this network between clusters, but with dedicated subnets per cluster. Several or all cluster nodes must be connected to this network. Traffic from external users to the externally available Kubernetes load-balanced services comes through the nodes that are connected to this network.

    • Services subnet(s)

      Provides IP addresses for externally available Kubernetes load-balanced services. The address ranges for MetalLB are assigned from this subnet. There can be several subnets per managed cluster that define the address ranges or address pools for MetalLB.

    • External subnet(s)

      Provides IP addresses that are statically allocated by the IPAM service to nodes. The IP gateway in this network is used as the default route on all nodes that are connected to this network. This network allows external users to connect to the cluster services exposed as Kubernetes load-balanced services. MetalLB speakers must run on the same nodes. For details, see Configure node selector for MetalLB speaker.

  • Storage network

    Serves storage access and replication traffic from and to Ceph OSD services. The storage network does not need to be connected to any IP routers and does not require external access, unless you want to use Ceph from outside of a Kubernetes cluster. To use a dedicated storage network, define and configure both subnets listed below.

    • Storage access subnet(s)

      Provides IP addresses that are statically allocated by the IPAM service to Ceph nodes. The Ceph OSD services bind to these addresses on their respective nodes. Serves Ceph access traffic from and to storage clients. This is a public network in Ceph terms. 1

    • Storage replication subnet(s)

      Provides IP addresses that are statically allocated by the IPAM service to Ceph nodes. The Ceph OSD services bind to these addresses on their respective nodes. Serves Ceph internal replication traffic. This is a cluster network in Ceph terms. 1

  • Out-of-Band (OOB) network

    Connects baseboard management controllers (BMCs) of the bare metal hosts. This network must not be accessible from the managed clusters.

The following diagram illustrates the networking schema of the Container Cloud deployment on bare metal with a managed cluster:

_images/bm-cluster-l3-networking-multihomed.png
1(1,2)

For more details about Ceph networks, see Ceph Network Configuration Reference.

Host networking

The following network roles are defined for all Mirantis Container Cloud clusters nodes on bare metal including the bootstrap, management, regional, and managed cluster nodes:

  • Out-of-band (OOB) network

    Connects the Baseboard Management Controllers (BMCs) of the hosts in the network to Ironic. This network is out of band for the host operating system.

  • PXE network

    Enables remote booting of servers through the PXE protocol. In management or regional clusters, DHCP server listens on this network for hosts discovery and inspection. In managed clusters, hosts use this network for the initial PXE boot and provisioning.

  • LCM network

    Connects LCM Agents running on the node to the LCM API of the management or regional cluster. It is also used for communication between kubelet and the Kubernetes API server inside a Kubernetes cluster. The MKE components use this network for communication inside a swarm cluster. In management or regional clusters, it is replaced by the management network.

  • Kubernetes workloads (pods) network

    Technology Preview

    Serves connections between Kubernetes pods. Each host has an address on this network, and this address is used by Calico as an endpoint to the underlay network.

  • Kubernetes external network

    Technology Preview

    Serves external connection to the Kubernetes API and the user services exposed by the cluster. In management or regional clusters, it is replaced by the management network.

  • Management network

    Serves external connections to the Container Cloud Management API and services of the management or regional cluster. Not available in a managed cluster.

  • Storage access network

    Connects Ceph nodes to the storage clients. The Ceph OSD service is bound to the address on this network. This is a public network in Ceph terms. 0

  • Storage replication network

    Connects Ceph nodes to each other. Serves internal replication traffic. This is a cluster network in Ceph terms. 0

Each network is represented on the host by a virtual Linux bridge. Physical interfaces may be connected to one of the bridges directly, or through a logical VLAN subinterface, or combined into a bond interface that is in turn connected to a bridge.

The following table summarizes the default names used for the bridges connected to the networks listed above:

Management or regional cluster

Network type

Bridge name

Assignment method TechPreview

OOB network

N/A

N/A

PXE network

bm-pxe

By a static interface name

Management network

k8s-lcm 2

By a subnet label ipam/SVC-k8s-lcm

Kubernetes workloads network

k8s-pods 1

By a static interface name

Managed cluster

Network type

Bridge name

Assignment method

OOB network

N/A

N/A

PXE network

N/A

N/A

LCM network

k8s-lcm 2

By a subnet label ipam/SVC-k8s-lcm

Kubernetes workloads network

k8s-pods 1

By a static interface name

Kubernetes external network

k8s-ext

By a static interface name

Storage access (public) network

ceph-public

By the subnet label ipam/SVC-ceph-public

Storage replication (cluster) network

ceph-cluster

By the subnet label ipam/SVC-ceph-cluster

0(1,2)

Ceph network configuration reference

1(1,2)

Interface name for this network role is static and cannot be changed.

2(1,2)

Use of this interface name (and network role) is mandatory for every cluster.

Storage

The baremetal-based Mirantis Container Cloud uses Ceph as a distributed storage system for file, block, and object storage. This section provides an overview of a Ceph cluster deployed by Container Cloud.

Overview

Mirantis Container Cloud deploys Ceph on baremetal-based managed clusters using Helm charts with the following components:

Rook Ceph Operator

A storage orchestrator that deploys Ceph on top of a Kubernetes cluster. Also known as Rook or Rook Operator. Rook operations include:

  • Deploying and managing a Ceph cluster based on provided Rook CRs such as CephCluster, CephBlockPool, CephObjectStore, and so on.

  • Orchestrating the state of the Ceph cluster and all its daemons.

KaaSCephCluster custom resource (CR)

Represents the customization of a Kubernetes installation and allows you to define the required Ceph configuration through the Container Cloud web UI before deployment. For example, you can define the failure domain, Ceph pools, Ceph node roles, number of Ceph components such as Ceph OSDs, and so on. The ceph-kcc-controller controller on the Container Cloud management cluster manages the KaaSCephCluster CR.

Ceph Controller

A Kubernetes controller that obtains the parameters from Container Cloud through a CR, creates CRs for Rook and updates its CR status based on the Ceph cluster deployment progress. It creates users, pools, and keys for OpenStack and Kubernetes and provides Ceph configurations and keys to access them. Also, Ceph Controller eventually obtains the data from the OpenStack Controller for the Keystone integration and updates the RADOS Gateway services configurations to use Kubernetes for user authentication. Ceph Controller operations include:

  • Transforming user parameters from the Container Cloud Ceph CR into Rook CRs and deploying a Ceph cluster using Rook.

  • Providing integration of the Ceph cluster with Kubernetes.

  • Providing data for OpenStack to integrate with the deployed Ceph cluster.

Ceph Status Controller

A Kubernetes controller that collects all valuable parameters from the current Ceph cluster, its daemons, and entities and exposes them into the KaaSCephCluster status. Ceph Status Controller operations include:

  • Collecting all statuses from a Ceph cluster and corresponding Rook CRs.

  • Collecting additional information on the health of Ceph daemons.

  • Provides information to the status section of the KaaSCephCluster CR.

Ceph Request Controller

A Kubernetes controller that obtains the parameters from Container Cloud through a CR and manages Ceph OSD lifecycle management (LCM) operations. It allows for a safe Ceph OSD removal from the Ceph cluster. Ceph Request Controller operations include:

  • Providing an ability to perform Ceph OSD LCM operations.

  • Obtaining specific CRs to remove Ceph OSDs and executing them.

  • Pausing the regular Ceph Controller reconcile until all requests are completed.

A typical Ceph cluster consists of the following components:

  • Ceph Monitors - three or, in rare cases, five Ceph Monitors.

  • Ceph Managers:

    • Before Container Cloud 2.22.0, one Ceph Manager.

    • Since Container Cloud 2.22.0, two Ceph Managers.

  • RADOS Gateway services - Mirantis recommends having three or more RADOS Gateway instances for HA.

  • Ceph OSDs - the number of Ceph OSDs may vary according to the deployment needs.

    Warning

    • A Ceph cluster with 3 Ceph nodes does not provide hardware fault tolerance and is not eligible for recovery operations, such as a disk or an entire Ceph node replacement.

    • A Ceph cluster uses the replication factor that equals 3. If the number of Ceph OSDs is less than 3, a Ceph cluster moves to the degraded state with the write operations restriction until the number of alive Ceph OSDs equals the replication factor again.

The placement of Ceph Monitors and Ceph Managers is defined in the KaaSCephCluster CR.

The following diagram illustrates the way a Ceph cluster is deployed in Container Cloud:

_images/ceph-deployment.png

The following diagram illustrates the processes within a deployed Ceph cluster:

_images/ceph-data-flow.png
Limitations

A Ceph cluster configuration in Mirantis Container Cloud includes but is not limited to the following limitations:

  • Only one Ceph Controller per a managed cluster and only one Ceph cluster per Ceph Controller are supported.

  • The replication size for any Ceph pool must be set to more than 1.

  • Only one CRUSH tree per cluster. The separation of devices per Ceph pool is supported through device classes with only one pool of each type for a device class.

  • Only the following types of CRUSH buckets are supported:

    • topology.kubernetes.io/region

    • topology.kubernetes.io/zone

    • topology.rook.io/datacenter

    • topology.rook.io/room

    • topology.rook.io/pod

    • topology.rook.io/pdu

    • topology.rook.io/row

    • topology.rook.io/rack

    • topology.rook.io/chassis

  • Only IPv4 is supported.

  • If two or more Ceph OSDs are located on the same device, there must be no dedicated WAL or DB for this class.

  • Only a full collocation or dedicated WAL and DB configurations are supported.

  • The minimum size of any defined Ceph OSD device is 5 GB.

  • Reducing the number of Ceph Monitors is not supported and causes the Ceph Monitor daemons removal from random nodes.

  • Removal of the mgr role in the nodes section of the KaaSCephCluster CR does not remove Ceph Managers. To remove a Ceph Manager from a node, remove it from the nodes spec and manually delete the mgr pod in the Rook namespace.

  • When adding a Ceph node with the Ceph Monitor role, if any issues occur with the Ceph Monitor, rook-ceph removes it and adds a new Ceph Monitor instead, named using the next alphabetic character in order. Therefore, the Ceph Monitor names may not follow the alphabetical order. For example, a, b, d, instead of a, b, c.

  • Ceph cluster does not support removable devices (with hotplug enabled) for deploying Ceph OSDs. This limitation has been lifted since Cluster releases 14.0.1 and 15.0.1 delivered in Container Cloud 2.24.2.

  • Ceph OSDs support only raw disks as data devices meaning that no dm or lvm devices are allowed.

  • Ceph does not support allocation of Ceph RGW pods on nodes where the Federal Information Processing Standard (FIPS) mode is enabled.

Addressing storage devices

There are several formats to use when specifying and addressing storage devices of a Ceph cluster. The default and recommended one is the /dev/disk/by-id format. This format is reliable and unaffected by the disk controller actions, such as device name shuffling or /dev/disk/by-path recalculating.

Difference between by-id, name, and by-path formats

The storage device /dev/disk/by-id format in most of the cases bases on a disk serial number, which is unique for each disk. A by-id symlink is created by the udev rules in the following format, where <BusID> is an ID of the bus to which the disk is attached and <DiskSerialNumber> stands for a unique disk serial number:

/dev/disk/by-id/<BusID>-<DiskSerialNumber>

Typical by-id symlinks for storage devices look as follows:

/dev/disk/by-id/nvme-SAMSUNG_MZ1LB3T8HMLA-00007_S46FNY0R394543
/dev/disk/by-id/scsi-SATA_HGST_HUS724040AL_PN1334PEHN18ZS
/dev/disk/by-id/ata-WDC_WD4003FZEX-00Z4SA0_WD-WMC5D0D9DMEH

In the example above, symlinks contain the following IDs:

  • Bus IDs: nvme, scsi-SATA and ata

  • Disk serial numbers: SAMSUNG_MZ1LB3T8HMLA-00007_S46FNY0R394543, HGST_HUS724040AL_PN1334PEHN18ZS and WDC_WD4003FZEX-00Z4SA0_WD-WMC5D0D9DMEH.

An exception to this rule is the wwn by-id symlinks, which are programmatically generated at boot. They are not solely based on disk serial numbers but also include other node information. This can lead to the wwn being recalculated when the node reboots. As a result, this symlink type cannot guarantee a persistent disk identifier and should not be used as a stable storage device symlink in a Ceph cluster.

The storage device name and by-path formats cannot be considered persistent because the sequence in which block devices are added during boot is semi-arbitrary. This means that block device names, for example, nvme0n1 and sdc, are assigned to physical disks during discovery, which may vary inconsistently from the previous node state. The same inconsistency applies to by-path symlinks, as they rely on the shortest physical path to the device at boot and may differ from the previous node state.

Therefore, Mirantis highly recommends using storage device by-id symlinks that contain disk serial numbers. This approach enables you to use a persistent device identifier addressed in the Ceph cluster specification.

Example KaaSCephCluster with device by-id identifiers

Below is an example KaaSCephCluster custom resource using the /dev/disk/by-id format for storage devices specification:

Note

Container Cloud enables you to use fullPath for the by-id symlinks since 2.25.0. For the earlier product versions, use the name field instead.

 apiVersion: kaas.mirantis.com/v1alpha1
 kind: KaaSCephCluster
 metadata:
   name: ceph-cluster-managed-cluster
   namespace: managed-ns
 spec:
   cephClusterSpec:
     nodes:
       # Add the exact ``nodes`` names.
       # Obtain the name from the "get machine" list.
       cz812-managed-cluster-storage-worker-noefi-58spl:
         roles:
         - mgr
         - mon
       # All disk configuration must be reflected in ``status.providerStatus.hardware.storage`` of the ``Machine`` object
         storageDevices:
         - config:
             deviceClass: ssd
           fullPath: /dev/disk/by-id/scsi-1ATA_WDC_WDS100T2B0A-00SM50_200231440912
       cz813-managed-cluster-storage-worker-noefi-lr4k4:
         roles:
         - mgr
         - mon
         storageDevices:
         - config:
             deviceClass: nvme
           fullPath: /dev/disk/by-id/nvme-SAMSUNG_MZ1LB3T8HMLA-00007_S46FNY0R394543
       cz814-managed-cluster-storage-worker-noefi-z2m67:
         roles:
         - mgr
         - mon
         storageDevices:
         - config:
             deviceClass: nvme
           fullPath: /dev/disk/by-id/nvme-SAMSUNG_ML1EB3T8HMLA-00007_S46FNY1R130423
     pools:
     - default: true
       deviceClass: ssd
       name: kubernetes
       replicated:
         size: 3
       role: kubernetes
   k8sCluster:
     name: managed-cluster
     namespace: managed-ns
Extended hardware configuration

Mirantis Container Cloud provides APIs that enable you to define hardware configurations that extend the reference architecture:

  • Bare Metal Host Profile API

    Enables for quick configuration of host boot and storage devices and assigning of custom configuration profiles to individual machines. See Create a custom bare metal host profile.

  • IP Address Management API

    Enables for quick configuration of host network interfaces and IP addresses and setting up of IP addresses ranges for automatic allocation. See Create L2 templates.

Typically, operations with the extended hardware configurations are available through the API and CLI, but not the web UI.

Automatic upgrade of a host operating system

To keep operating system on a bare metal host up to date with the latest security updates, the operating system requires periodic software packages upgrade that may or may not require the host reboot.

Mirantis Container Cloud uses life cycle management tools to update the operating system packages on the bare metal hosts. Container Cloud may also trigger restart of bare metal hosts to apply the updates.

In the management cluster of Container Cloud, software package upgrade and host restart is applied automatically when a new Container Cloud version with available kernel or software packages upgrade is released.

In managed clusters, package upgrade and host restart is applied as part of usual cluster upgrade using the Update cluster option in the Container Cloud web UI.

Operating system upgrade and host restart are applied to cluster nodes one by one. If Ceph is installed in the cluster, the Container Cloud orchestration securely pauses the Ceph OSDs on the node before restart. This allows avoiding degradation of the storage service.

Caution

  • Depending on the cluster configuration, applying security updates and host restart can increase the update time for each node to up to 1 hour.

  • Cluster nodes are updated one by one. Therefore, for large clusters, the update may take several days to complete.

Built-in load balancing

The Mirantis Container Cloud managed clusters that are based on vSphere or bare metal use MetalLB for load balancing of services and HAProxy with VIP managed by Virtual Router Redundancy Protocol (VRRP) with Keepalived for the Kubernetes API load balancer.

Kubernetes API load balancing

Every control plane node of each Kubernetes cluster runs the kube-api service in a container. This service provides a Kubernetes API endpoint. Every control plane node also runs the haproxy server that provides load balancing with back-end health checking for all kube-api endpoints as back ends.

The default load balancing method is least_conn. With this method, a request is sent to the server with the least number of active connections. The default load balancing method cannot be changed using the Container Cloud API.

Only one of the control plane nodes at any given time serves as a front end for Kubernetes API. To ensure this, the Kubernetes clients use a virtual IP (VIP) address for accessing Kubernetes API. This VIP is assigned to one node at a time using VRRP. Keepalived running on each control plane node provides health checking and failover of the VIP.

Keepalived is configured in multicast mode.

Note

The use of VIP address for load balancing of Kubernetes API requires that all control plane nodes of a Kubernetes cluster are connected to a shared L2 segment. This limitation prevents from installing full L3 topologies where control plane nodes are split between different L2 segments and L3 networks.

Caution

External load balancers for services are not supported by the current version of the Container Cloud vSphere provider. The built-in load balancing described in this section is the only supported option and cannot be disabled.

Services load balancing

The services provided by the Kubernetes clusters, including Container Cloud and user services, are balanced by MetalLB. The metallb-speaker service runs on every worker node in the cluster and handles connections to the service IP addresses.

MetalLB runs in the MAC-based (L2) mode. It means that all control plane nodes must be connected to a shared L2 segment. This is a limitation that does not allow installing full L3 cluster topologies.

Caution

External load balancers for services are not supported by the current version of the Container Cloud vSphere provider. The built-in load balancing described in this section is the only supported option and cannot be disabled.

VMware vSphere network objects and IPAM recommendations

The VMware vSphere provider of Mirantis Container Cloud supports the following types of vSphere network objects:

  • Virtual network

    A network of virtual machines running on a hypervisor(s) that are logically connected to each other so that they can exchange data. Virtual machines can be connected to virtual networks that you create when you add a network.

  • Distributed port group

    A port group associated with a vSphere distributed switch that specifies port configuration options for each member port. Distributed port groups define how connection is established through the vSphere distributed switch to the network.

A Container Cloud cluster can be deployed using one of these network objects with or without a DHCP server in the network:

  • Non-DHCP

    Container Cloud uses IPAM service to manage IP addresses assignment to machines. You must provide additional network parameters, such as CIDR, gateway, IP ranges, and nameservers. Container Cloud processes this data to the cloud-init metadata and passes the data to machines during their bootstrap.

  • DHCP

    Container Cloud relies on a DHCP server to assign IP addresses to virtual machines.

Mirantis recommends using IP address management (IPAM) for cluster machines provided by Container Cloud. IPAM must be enabled for deployment in the non-DHCP vSphere networks. But Mirantis recommends enabling IPAM in the DHCP-based networks as well. In this case, the dedicated IPAM range should not intersect with the IP range used in the DHCP server configuration for the provided vSphere network. Such configuration prevents issues with accidental IP address change for machines. For the issue details, see vSphere troubleshooting.

Note

To obtain IPAM parameters for the selected vSphere network, contact your vSphere administrator who provides you with IP ranges dedicated to your environment only.

The following parameters are required to enable IPAM:

  • Network CIDR.

  • Network gateway address.

  • Minimum 1 DNS server.

  • IP address include range to be allocated for cluster machines. Make sure that this range is not part of the DHCP range if the network has a DHCP server.

    Minimal number of addresses in the range:

    • 3 IPs for management or regional cluster

    • 3+N IPs for a managed cluster, where N is the number of worker nodes

  • Optional. IP address exclude range that is the list of IPs not to be assigned to machines from the include ranges.

A dedicated Container Cloud network must not contain any virtual machines with the keepalived instance running inside them as this may lead to the vrouter_id conflict. By default, the Container Cloud management or regional cluster is deployed with vrouter_id set to 1. Managed clusters are deployed with the vrouter_id value starting from 2 and upper.

Kubernetes lifecycle management

The Kubernetes lifecycle management (LCM) engine in Mirantis Container Cloud consists of the following components:

LCM Controller

Responsible for all LCM operations. Consumes the LCMCluster object and orchestrates actions through LCM Agent.

LCM Agent

Runs on the target host. Executes Ansible playbooks in headless mode. Does not run on attached MKE clusters that are not originally deployed by Container Cloud.

Helm Controller

Responsible for the Helm charts life cycle, is installed by a cloud provider as a Helm v3 chart.

The Kubernetes LCM components handle the following custom resources:

  • LCMCluster

  • LCMMachine

  • HelmBundle

The following diagram illustrates handling of the LCM custom resources by the Kubernetes LCM components. On a managed cluster, apiserver handles multiple Kubernetes objects, for example, deployments, nodes, RBAC, and so on.

_images/lcm-components.png
LCM custom resources

The Kubernetes LCM components handle the following custom resources (CRs):

  • LCMMachine

  • LCMCluster

  • HelmBundle

LCMMachine

Describes a machine that is located on a cluster. It contains the machine type, control or worker, StateItems that correspond to Ansible playbooks and miscellaneous actions, for example, downloading a file or executing a shell command. LCMMachine reflects the current state of the machine, for example, a node IP address, and each StateItem through its status. Multiple LCMMachine CRs can correspond to a single cluster.

LCMCluster

Describes a managed cluster. In its spec, LCMCluster contains a set of StateItems for each type of LCMMachine, which describe the actions that must be performed to deploy the cluster. LCMCluster is created by the provider, using machineTypes of the Release object. The status field of LCMCluster reflects the status of the cluster, for example, the number of ready or requested nodes.

HelmBundle

Wrapper for Helm charts that is handled by Helm Controller. HelmBundle tracks what Helm charts must be installed on a managed cluster.

LCM Controller

LCM Controller runs on the management and regional cluster and orchestrates the LCMMachine objects according to their type and their LCMCluster object.

Once the LCMCluster and LCMMachine objects are created, LCM Controller starts monitoring them to modify the spec fields and update the status fields of the LCMMachine objects when required. The status field of LCMMachine is updated by LCM Agent running on a node of a management, regional, or managed cluster.

Each LCMMachine has the following lifecycle states:

  1. Uninitialized - the machine is not yet assigned to an LCMCluster.

  2. Pending - the agent reports a node IP address and host name.

  3. Prepare - the machine executes StateItems that correspond to the prepare phase. This phase usually involves downloading the necessary archives and packages.

  4. Deploy - the machine executes StateItems that correspond to the deploy phase that is becoming a Mirantis Kubernetes Engine (MKE) node.

  5. Ready - the machine is being deployed.

  6. Upgrade - the machine is being upgraded to the new MKE version.

  7. Reconfigure - the machine executes StateItems that correspond to the reconfigure phase. The machine configuration is being updated without affecting workloads running on the machine.

The templates for StateItems are stored in the machineTypes field of an LCMCluster object, with separate lists for the MKE manager and worker nodes. Each StateItem has the execution phase field for a management, regional, and managed cluster:

  1. The prepare phase is executed for all machines for which it was not executed yet. This phase comprises downloading the files necessary for the cluster deployment, installing the required packages, and so on.

  2. During the deploy phase, a node is added to the cluster. LCM Controller applies the deploy phase to the nodes in the following order:

    1. First manager node is deployed.

    2. The remaining manager nodes are deployed one by one and the worker nodes are deployed in batches (by default, up to 50 worker nodes at the same time).

LCM Controller deploys and upgrades a Mirantis Container Cloud cluster by setting StateItems of LCMMachine objects following the corresponding StateItems phases described above. The Container Cloud cluster upgrade process follows the same logic that is used for a new deployment, that is applying a new set of StateItems to the LCMMachines after updating the LCMCluster object. But if the existing worker node is being upgraded, LCM Controller performs draining and cordoning on this node honoring the Pod Disruption Budgets. This operation prevents unexpected disruptions of the workloads.

LCM Agent

LCM Agent handles a single machine that belongs to a management, regional, or managed cluster. It runs on the machine operating system but communicates with apiserver of the regional cluster. LCM Agent is deployed as a systemd unit using cloud-init. LCM Agent has a built-in self-upgrade mechanism.

LCM Agent monitors the spec of a particular LCMMachine object to reconcile the machine state with the object StateItems and update the LCMMachine status accordingly. The actions that LCM Agent performs while handling the StateItems are as follows:

  • Download configuration files

  • Run shell commands

  • Run Ansible playbooks in headless mode

LCM Agent provides the IP address and hostname of the machine for the LCMMachine status parameter.

Helm Controller

Helm Controller is used by Mirantis Container Cloud to handle management, regional, and managed clusters core addons such as StackLight and the application addons such as the OpenStack components.

Helm Controller is installed as a separate Helm v3 chart by the Container Cloud provider. Its Pods are created using Deployment.

The Helm release information is stored in the KaaSRelease object for the management and regional clusters and in the ClusterRelease object for all types of the Container Cloud clusters. These objects are used by the Container Cloud provider. The Container Cloud provider uses the information from the ClusterRelease object together with the Container Cloud API Cluster spec. In Cluster spec, the operator can specify the Helm release name and charts to use. By combining the information from the Cluster providerSpec parameter and its ClusterRelease object, the cluster actuator generates the LCMCluster objects. These objects are further handled by LCM Controller and the HelmBundle object handled by Helm Controller. HelmBundle must have the same name as the LCMCluster object for the cluster that HelmBundle applies to.

Although a cluster actuator can only create a single HelmBundle per cluster, Helm Controller can handle multiple HelmBundle objects per cluster.

Helm Controller handles the HelmBundle objects and reconciles them with the state of Helm in its cluster.

Helm Controller can also be used by the management cluster with corresponding HelmBundle objects created as part of the initial management cluster setup.

Identity and access management

Identity and access management (IAM) provides a central point of users and permissions management of the Mirantis Container Cloud cluster resources in a granular and unified manner. Also, IAM provides infrastructure for single sign-on user experience across all Container Cloud web portals.

IAM for Container Cloud consists of the following components:

Keycloak
  • Provides the OpenID Connect endpoint

  • Integrates with an external identity provider (IdP), for example, existing LDAP or Google Open Authorization (OAuth)

  • Stores roles mapping for users

IAM Controller
  • Provides IAM API with data about Container Cloud projects

  • Handles all role-based access control (RBAC) components in Kubernetes API

IAM API

Provides an abstraction API for creating user scopes and roles

External identity provider integration

To be consistent and keep the integrity of a user database and user permissions, in Mirantis Container Cloud, IAM stores the user identity information internally. However in real deployments, the identity provider usually already exists.

Out of the box, in Container Cloud, IAM supports integration with LDAP and Google Open Authorization (OAuth). If LDAP is configured as an external identity provider, IAM performs one-way synchronization by mapping attributes according to configuration.

In the case of the Google Open Authorization (OAuth) integration, the user is automatically registered and their credentials are stored in the internal database according to the user template configuration. The Google OAuth registration workflow is as follows:

  1. The user requests a Container Cloud web UI resource.

  2. The user is redirected to the IAM login page and logs in using the Log in with Google account option.

  3. IAM creates a new user with the default access rights that are defined in the user template configuration.

  4. The user can access the Container Cloud web UI resource.

The following diagram illustrates the external IdP integration to IAM:

_images/iam-ext-idp.png

You can configure simultaneous integration with both external IdPs with the user identity matching feature enabled.

Authentication and authorization

Mirantis IAM uses the OpenID Connect (OIDC) protocol for handling authentication.

Implementation flow

Mirantis IAM performs as an OpenID Connect (OIDC) provider, it issues a token and exposes discovery endpoints.

The credentials can be handled by IAM itself or delegated to an external identity provider (IdP).

The issued JSON Web Token (JWT) is sufficient to perform operations across Mirantis Container Cloud according to the scope and role defined in it. Mirantis recommends using asymmetric cryptography for token signing (RS256) to minimize the dependency between IAM and managed components.

When Container Cloud calls Mirantis Kubernetes Engine (MKE), the user in Keycloak is created automatically with a JWT issued by Keycloak on behalf of the end user. MKE, in its turn, verifies whether the JWT is issued by Keycloak. If the user retrieved from the token does not exist in the MKE database, the user is automatically created in the MKE database based on the information from the token.

The authorization implementation is out of the scope of IAM in Container Cloud. This functionality is delegated to the component level. IAM interacts with a Container Cloud component using the OIDC token content that is processed by a component itself and required authorization is enforced. Such an approach enables you to have any underlying authorization that is not dependent on IAM and still to provide a unified user experience across all Container Cloud components.

Kubernetes CLI authentication flow

The following diagram illustrates the Kubernetes CLI authentication flow. The authentication flow for Helm and other Kubernetes-oriented CLI utilities is identical to the Kubernetes CLI flow, but JSON Web Tokens (JWT) must be pre-provisioned.

_images/iam-authn-k8s.png

See also

IAM resources

Monitoring

Mirantis Container Cloud uses StackLight, the logging, monitoring, and alerting solution that provides a single pane of glass for cloud maintenance and day-to-day operations as well as offers critical insights into cloud health including operational information about the components deployed in management, regional, and managed clusters. StackLight is based on Prometheus, an open-source monitoring solution and a time series database.

Deployment architecture

Mirantis Container Cloud deploys the StackLight stack as a release of a Helm chart that contains the helm-controller and helmbundles.lcm.mirantis.com (HelmBundle) custom resources. The StackLight HelmBundle consists of a set of Helm charts with the StackLight components that include:

StackLight components overview

StackLight component

Description

Alerta

Receives, consolidates, and deduplicates the alerts sent by Alertmanager and visually represents them through a simple web UI. Using the Alerta web UI, you can view the most recent or watched alerts, group, and filter alerts.

Alertmanager

Handles the alerts sent by client applications such as Prometheus, deduplicates, groups, and routes alerts to receiver integrations. Using the Alertmanager web UI, you can view the most recent fired alerts, silence them, or view the Alertmanager configuration.

Elasticsearch Curator

Maintains the data (indexes) in OpenSearch by performing such operations as creating, closing, or opening an index as well as deleting a snapshot. Also, manages the data retention policy in OpenSearch.

Elasticsearch Exporter Compatible with OpenSearch

The Prometheus exporter that gathers internal OpenSearch metrics.

Grafana

Builds and visually represents metric graphs based on time series databases. Grafana supports querying of Prometheus using the PromQL language.

Database back ends

StackLight uses PostgreSQL for Alerta and Grafana. PostgreSQL reduces the data storage fragmentation while enabling high availability. High availability is achieved using Patroni, the PostgreSQL cluster manager that monitors for node failures and manages failover of the primary node. StackLight also uses Patroni to manage major version upgrades of PostgreSQL clusters, which allows leveraging the database engine functionality and improvements as they are introduced upstream in new releases, maintaining functional continuity without version lock-in.

Logging stack

Responsible for collecting, processing, and persisting logs and Kubernetes events. By default, when deploying through the Container Cloud web UI, only the metrics stack is enabled on managed clusters. To enable StackLight to gather managed cluster logs, enable the logging stack during deployment. On management clusters, the logging stack is enabled by default. The logging stack components include:

  • OpenSearch, which stores logs and notifications.

  • Fluentd-logs, which collects logs, sends them to OpenSearch, generates metrics based on analysis of incoming log entries, and exposes these metrics to Prometheus.

  • OpenSearch Dashboards, which provides real-time visualization of the data stored in OpenSearch and enables you to detect issues.

  • Metricbeat, which collects Kubernetes events and sends them to OpenSearch for storage.

  • Prometheus-es-exporter, which presents the OpenSearch data as Prometheus metrics by periodically sending configured queries to the OpenSearch cluster and exposing the results to a scrapable HTTP endpoint like other Prometheus targets.

Note

The logging mechanism performance depends on the cluster log load. In case of a high load, you may need to increase the default resource requests and limits for fluentdLogs. For details, see StackLight configuration parameters: Resource limits.

Metric collector

Collects telemetry data (CPU or memory usage, number of active alerts, and so on) from Prometheus and sends the data to centralized cloud storage for further processing and analysis. Metric collector runs on the management cluster.

Note

This component is designated for internal StackLight use only.

Prometheus

Gathers metrics. Automatically discovers and monitors the endpoints. Using the Prometheus web UI, you can view simple visualizations and debug. By default, the Prometheus database stores metrics of the past 15 days or up to 15 GB of data depending on the limit that is reached first.

Prometheus Blackbox Exporter

Allows monitoring endpoints over HTTP, HTTPS, DNS, TCP, and ICMP.

Prometheus-es-exporter

Presents the OpenSearch data as Prometheus metrics by periodically sending configured queries to the OpenSearch cluster and exposing the results to a scrapable HTTP endpoint like other Prometheus targets.

Prometheus Node Exporter

Gathers hardware and operating system metrics exposed by kernel.

Prometheus Relay

Adds a proxy layer to Prometheus to merge the results from underlay Prometheus servers to prevent gaps in case some data is missing on some servers. Is available only in the HA StackLight mode.

Reference Application Available since 2.21.0

Enables workload monitoring on non-MOSK managed clusters. Mimics a classical microservice application and provides metrics that describe the likely behavior of user workloads.

Note

For the feature support on MOSK deployments, refer to MOSK documentation: Deploy RefApp using automation tools.

Salesforce notifier

Enables sending Alertmanager notifications to Salesforce to allow creating Salesforce cases and closing them once the alerts are resolved. Disabled by default.

Salesforce reporter

Queries Prometheus for the data about the amount of vCPU, vRAM, and vStorage used and available, combines the data, and sends it to Salesforce daily. Mirantis uses the collected data for further analysis and reports to improve the quality of customer support. Disabled by default.

Telegraf

Collects metrics from the system. Telegraf is plugin-driven and has the concept of two distinct set of plugins: input plugins collect metrics from the system, services, or third-party APIs; output plugins write and expose metrics to various destinations.

The Telegraf agents used in Container Cloud include:

  • telegraf-ds-smart monitors SMART disks, and runs on both management and managed clusters.

  • telegraf-ironic monitors Ironic on the baremetal-based management clusters. The ironic input plugin collects and processes data from Ironic HTTP API, while the http_response input plugin checks Ironic HTTP API availability. As an output plugin, to expose collected data as Prometheus target, Telegraf uses prometheus.

  • telegraf-docker-swarm gathers metrics from the Mirantis Container Runtime API about the Docker nodes, networks, and Swarm services. This is a Docker Telegraf input plugin with downstream additions.

Telemeter

Enables a multi-cluster view through a Grafana dashboard of the management cluster. Telemeter includes a Prometheus federation push server and clients to enable isolated Prometheus instances, which cannot be scraped from a central Prometheus instance, to push metrics to the central location.

The Telemeter services are distributed as follows:

  • Management cluster hosts the Telemeter server

  • Regional clusters host the Telemeter server and Telemeter client

  • Managed clusters host the Telemeter client

The metrics from managed clusters are aggregated on regional clusters. Then both regional and managed clusters metrics are sent from regional clusters to the management cluster.

Note

This component is designated for internal StackLight use only.

Every Helm chart contains a default values.yml file. These default values are partially overridden by custom values defined in the StackLight Helm chart.

Before deploying a managed cluster, you can select the HA or non-HA StackLight architecture type. The non-HA mode is set by default. On the management and regional clusters, StackLight is deployed in the HA mode only. The following table lists the differences between the HA and non-HA modes:

StackLight database modes

Non-HA StackLight mode default

HA StackLight mode

  • One Prometheus instance

  • One Alertmanager instance Since 2.24.0 and 2.24.2 for MOSK 23.2

  • One OpenSearch instance

  • One PostgreSQL instance

  • One iam-proxy instance

One persistent volume is provided for storing data. In case of a service or node failure, a new pod is redeployed and the volume is reattached to provide the existing data. Such setup has a reduced hardware footprint but provides less performance.

  • Two Prometheus instances

  • Two Alertmanager instances

  • Three OpenSearch instances

  • Three PostgreSQL instances

  • Two iam-proxy instances Since 2.23.0 and 2.23.1 for MOSK 23.1

Local Volume Provisioner is used to provide local host storage. In case of a service or node failure, the traffic is automatically redirected to any other running Prometheus or OpenSearch server. For better performance, Mirantis recommends that you deploy StackLight in the HA mode. Two iam-proxy instances ensure access to HA components if one iam-proxy node fails.

Note

Before Container Cloud 2.24.0, Alertmanager has 2 replicas in the non-HA mode.

Depending on the Container Cloud cluster type and selected StackLight database mode, StackLight is deployed on the following number of nodes:

StackLight database modes

Cluster

StackLight database mode

Target nodes

Management and regional

HA mode

All Kubernetes master nodes

Managed

Non-HA mode

  • All nodes with the stacklight label.

  • If no nodes have the stacklight label, StackLight is spread across all worker nodes. The minimal requirement is at least 1 worker node.

HA mode

All nodes with the stacklight label. The minimal requirement is 3 nodes with the stacklight label. Otherwise, StackLight deployment does not start.

Authentication flow

StackLight provides five web UIs including Prometheus, Alertmanager, Alerta, OpenSearch Dashboards, and Grafana. Access to StackLight web UIs is protected by Keycloak-based Identity and access management (IAM). All web UIs except Alerta are exposed to IAM through the IAM proxy middleware. The Alerta configuration provides direct integration with IAM.

The following diagram illustrates accessing the IAM-proxied StackLight web UIs, for example, Prometheus web UI:

_images/sl-auth-iam-proxied.png

Authentication flow for the IAM-proxied StackLight web UIs:

  1. A user enters the public IP of a StackLight web UI, for example, Prometheus web UI.

  2. The public IP leads to IAM proxy, deployed as a Kubernetes LoadBalancer, which protects the Prometheus web UI.

  3. LoadBalancer routes the HTTP request to Kubernetes internal IAM proxy service endpoints, specified in the X-Forwarded-Proto or X-Forwarded-Host headers.

  4. The Keycloak login form opens (the login_url field in the IAM proxy configuration, which points to Keycloak realm) and the user enters the user name and password.

  5. Keycloak validates the user name and password.

  6. The user obtains access to the Prometheus web UI (the upstreams field in the IAM proxy configuration).

Note

  • The discovery URL is the URL of the IAM service.

  • The upstream URL is the hidden endpoint of a web UI (Prometheus web UI in the example above).

The following diagram illustrates accessing the Alerta web UI:

_images/sl-authentication-direct.png

Authentication flow for the Alerta web UI:

  1. A user enters the public IP of the Alerta web UI.

  2. The public IP leads to Alerta deployed as a Kubernetes LoadBalancer type.

  3. LoadBalancer routes the HTTP request to the Kubernetes internal Alerta service endpoint.

  4. The Keycloak login form opens (Alerta refers to the IAM realm) and the user enters the user name and password.

  5. Keycloak validates the user name and password.

  6. The user obtains access to the Alerta web UI.

Supported features

Using the Mirantis Container Cloud web UI, on the pre-deployment stage of a managed cluster, you can view, enable or disable, or tune the following StackLight features available:

  • StackLight HA mode.

  • Database retention size and time for Prometheus.

  • Tunable index retention period for OpenSearch.

  • Tunable PersistentVolumeClaim (PVC) size for Prometheus and OpenSearch set to 16 GB for Prometheus and 30 GB for OpenSearch by default. The PVC size must be logically aligned with the retention periods or sizes for these components.

  • Email and Slack receivers for the Alertmanager notifications.

  • Predefined set of dashboards.

  • Predefined set of alerts and capability to add new custom alerts for Prometheus in the following exemplary format:

    - alert: HighErrorRate
      expr: job:request_latency_seconds:mean5m{job="myjob"} > 0.5
      for: 10m
      labels:
        severity: page
      annotations:
        summary: High request latency
    
Monitored components

StackLight measures, analyzes, and reports in a timely manner about failures that may occur in the following Mirantis Container Cloud components and their sub-components, if any:

  • Ceph

  • Ironic (Container Cloud bare-metal provider)

  • Kubernetes services:

    • Calico

    • etcd

    • Kubernetes cluster

    • Kubernetes containers

    • Kubernetes deployments

    • Kubernetes nodes

  • NGINX

  • Node hardware and operating system

  • PostgreSQL

  • StackLight:

    • Alertmanager

    • OpenSearch

    • Grafana

    • Prometheus

    • Prometheus Relay

    • Salesforce notifier

    • Telemeter

  • SSL certificates

  • Mirantis Kubernetes Engine (MKE)

    • Docker/Swarm metrics (through Telegraf)

    • Built-in MKE metrics

Outbound cluster metrics

The data collected and transmitted through an encrypted channel back to Mirantis provides our Customer Success Organization information to better understand the operational usage patterns our customers are experiencing as well as to provide feedback on product usage statistics to enable our product teams to enhance our products and services for our customers.

Before the Cluster releases 17.0.0, 16.0.0, and 14.1.0, the summary of all deployed Container Cloud configurations is collected. The data is anonymized from all sensitive information, such as IDs, IP addresses, passwords, private keys, and so on. Before the Cluster releases mentioned above, Mirantis collects information about the following Container Cloud objects, if any:

  • Cluster

  • Machine

  • MachinePool

  • MCCUpgrade

  • BareMetalHost

  • BareMetalHostProfile

  • IPAMHost

  • IPAddr

  • KaaSCephCluster

  • L2Template

  • Subnet

The node-level resource data are broken down into three broad categories: Cluster, Node, and Namespace. The telemetry data tracks Allocatable, Capacity, Limits, Requests, and actual Usage of node-level resources.

Terms explanation

Term

Definition

Allocatable

On a Kubernetes Node, the amount of compute resources that are available for pods

Capacity

The total number of available resources regardless of current consumption

Limits

Constraints imposed by Administrators

Requests

The resources that a given container application is requesting

Usage

The actual usage or consumption of a given resource

The full list of the outbound data includes:

From all Container Cloud managed clusters
  • cluster_alerts_firing Since 2.23.0

  • cluster_filesystem_size_bytes

  • cluster_filesystem_usage_bytes

  • cluster_filesystem_usage_ratio

  • cluster_master_nodes_total

  • cluster_nodes_total

  • cluster_persistentvolumeclaim_requests_storage_bytes

  • cluster_total_alerts_triggered

  • cluster_capacity_cpu_cores

  • cluster_capacity_memory_bytes

  • cluster_usage_cpu_cores

  • cluster_usage_memory_bytes

  • cluster_usage_per_capacity_cpu_ratio

  • cluster_usage_per_capacity_memory_ratio

  • cluster_worker_nodes_total

  • cluster_workload_pods_total Since 2.22.0

  • cluster_workload_containers_total Since 2.22.0

  • kaas_info

  • kaas_cluster_machines_ready_total

  • kaas_cluster_machines_requested_total

  • kaas_clusters

  • kaas_cluster_updating Since 2.21.0

  • kaas_license_expiry

  • kaas_machines_ready

  • kaas_machines_requested

  • kubernetes_api_availability

  • node_labels Since 2.24.0

  • mke_api_availability

  • mke_cluster_nodes_total

  • mke_cluster_containers_total

  • mke_cluster_vcpu_free

  • mke_cluster_vcpu_used

  • mke_cluster_vram_free

  • mke_cluster_vram_used

  • mke_cluster_vstorage_free

  • mke_cluster_vstorage_used

From Mirantis OpenStack for Kubernetes (MOSK) clusters only
  • openstack_cinder_api_latency_90

  • openstack_cinder_api_latency_99

  • openstack_cinder_api_status

  • openstack_cinder_availability

  • openstack_cinder_volumes_total

  • openstack_glance_api_status

  • openstack_glance_availability

  • openstack_glance_images_total

  • openstack_glance_snapshots_total

  • openstack_heat_availability

  • openstack_heat_stacks_total

  • openstack_host_aggregate_instances Removed in 2.24.0

  • openstack_host_aggregate_memory_used_ratio Removed in 2.24.0

  • openstack_host_aggregate_memory_utilisation_ratio Removed in 2.24.0

  • openstack_host_aggregate_cpu_utilisation_ratio Removed in 2.24.0

  • openstack_host_aggregate_vcpu_used_ratio Removed in 2.24.0

  • openstack_instance_availability

  • openstack_instance_create_end

  • openstack_instance_create_error

  • openstack_instance_create_start

  • openstack_keystone_api_latency_90

  • openstack_keystone_api_latency_99

  • openstack_keystone_api_status

  • openstack_keystone_availability

  • openstack_keystone_tenants_total

  • openstack_keystone_users_total

  • openstack_kpi_provisioning

  • openstack_lbaas_availability

  • openstack_mysql_flow_control

  • openstack_neutron_api_latency_90

  • openstack_neutron_api_latency_99

  • openstack_neutron_api_status

  • openstack_neutron_availability

  • openstack_neutron_lbaas_loadbalancers_total

  • openstack_neutron_networks_total

  • openstack_neutron_ports_total

  • openstack_neutron_routers_total

  • openstack_neutron_subnets_total

  • openstack_nova_all_compute_cpu_utilisation

  • openstack_nova_all_compute_mem_utilisation

  • openstack_nova_all_computes_total

  • openstack_nova_all_vcpus_total

  • openstack_nova_all_used_vcpus_total

  • openstack_nova_all_ram_total_gb

  • openstack_nova_all_used_ram_total_gb

  • openstack_nova_all_disk_total_gb

  • openstack_nova_all_used_disk_total_gb

  • openstack_nova_api_status

  • openstack_nova_availability

  • openstack_nova_compute_cpu_utilisation

  • openstack_nova_compute_mem_utilisation

  • openstack_nova_computes_total

  • openstack_nova_disk_total_gb

  • openstack_nova_instances_active_total

  • openstack_nova_ram_total_gb

  • openstack_nova_used_disk_total_gb

  • openstack_nova_used_ram_total_gb

  • openstack_nova_used_vcpus_total

  • openstack_nova_vcpus_total

  • openstack_rmq_message_deriv

  • openstack_quota_instances

  • openstack_quota_ram_gb

  • openstack_quota_vcpus

  • openstack_quota_volume_storage_gb

  • openstack_usage_instances

  • openstack_usage_ram_gb

  • openstack_usage_vcpus

  • openstack_usage_volume_storage_gb

  • osdpl_aodh_alarms Since MOSK 23.3

  • osdpl_cinder_zone_volumes Since MOSK 23.3

  • osdpl_neutron_availability_zone_info Since MOSK 23.3

  • osdpl_neutron_zone_routers Since MOSK 23.3

  • osdpl_nova_aggregate_hosts Since MOSK 23.3

  • osdpl_nova_availability_zone_info Since MOSK 23.3

  • osdpl_nova_availability_zone_instances Since MOSK 23.3

  • osdpl_nova_availability_zone_hosts Since MOSK 23.3

  • osdpl_version_info Since MOSK 23.3

  • tf_operator_info Since MOSK 23.3 for Tungsten Fabric

StackLight proxy

StackLight components, which require external access, automatically use the same proxy that is configured for Mirantis Container Cloud clusters. Therefore, you only need to configure proxy during deployment of your management, regional, or managed clusters. No additional actions are required to set up proxy for StackLight. For more details about implementation of proxy support in Container Cloud, see Proxy and cache support.

Note

Proxy handles only the HTTP and HTTPS traffic. Therefore, for clusters with limited or no Internet access, it is not possible to set up Alertmanager email notifications, which use SMTP, when proxy is used.

Proxy is used for the following StackLight components:

Component

Cluster type

Usage

Alertmanager

Any

As a default http_config for all HTTP-based receivers except the predefined HTTP-alerta and HTTP-salesforce. For these receivers, http_config is overridden on the receiver level.

Metric Collector

Management

To send outbound cluster metrics to Mirantis.

Salesforce notifier

Any

To send notifications to the Salesforce instance.

Salesforce reporter

Any

To send metric reports to the Salesforce instance.

Telemeter client

Regional

To send all metrics from the clusters of a region, including the managed and regional clusters, to the management cluster. Proxy is not used for the Telemeter client on managed clusters because managed clusters must have a direct access to their regional cluster.

Reference Application for workload monitoring

Available since 2.21.0 for non-MOSK managed clusters

Note

For the feature support on MOSK deployments, refer to MOSK documentation: Deploy RefApp using automation tools.

Reference Application is a small microservice application that enables workload monitoring on non-MOSK managed clusters. It mimics a classical microservice application and provides metrics that describe the likely behavior of user workloads.

The application consists of the following API and database services that allow putting simple records into the database through the API and retrieving them:

Reference Application API

Runs on StackLight nodes and provides API access to the database. Runs three API instances for high availability.

PostgreSQL Since Container Cloud 2.22.0

Runs on worker nodes and stores the data on an attached PersistentVolumeClaim (PVC). Runs three database instances for high availability.

Note

Before version 2.22.0, Container Cloud used MariaDB as the database management system instead of PostgreSQL.

StackLight queries the API measuring response times for each query. No caching is being done, so each API request must go to the database, allowing to verify the availability of a stateful workload on the cluster.

Reference Application requires the following resources on top of the main product requirements:

  • Up to 1 GiB of RAM per cluster

  • Up to 3 GiB of storage per cluster

The feature is disabled by default and can be enabled using the StackLight configuration manifest as described in StackLight configuration parameters: Reference Application.

Hardware and system requirements

Using Mirantis Container Cloud, you can deploy a Mirantis Kubernetes Engine (MKE) cluster on bare metal, OpenStack, or VMware vSphere cloud providers. Each cloud provider requires corresponding resources.

Requirements for a bootstrap node

A bootstrap node is necessary only to deploy the management cluster. When the bootstrap is complete, the bootstrap node can be redeployed and its resources can be reused for the managed cluster workloads.

The minimum reference system requirements of a baremetal-based bootstrap seed node are described in System requirements for the seed node. The minimum reference system requirements a bootstrap node for other supported Container Cloud providers are as follows:

  • Any local machine on Ubuntu 20.04 that requires access to the provider API with the following configuration:

    • 2 vCPUs

    • 4 GB of RAM

    • 5 GB of available storage

    • Docker version currently available for Ubuntu 20.04

  • Internet access for downloading of all required artifacts

Note

For the vSphere cloud provider, you can use RHEL of the following versions with the same system requirements as for Ubuntu:

  • Since Container Cloud 2.25.0, RHEL 8.7

  • Before Container Cloud 2.25.0, RHEL 7.9

Requirements for a baremetal-based cluster

If you use a firewall or proxy, make sure that the bootstrap, management, and regional clusters have access to the following IP ranges and domain names required for the Container Cloud content delivery network and alerting:

  • IP ranges:

  • Domain names:

    • mirror.mirantis.com and repos.mirantis.com for packages

    • binary.mirantis.com for binaries and Helm charts

    • mirantis.azurecr.io and *.blob.core.windows.net for Docker images

    • mcc-metrics-prod-ns.servicebus.windows.net:9093 for Telemetry (port 443 if proxy is enabled)

    • mirantis.my.salesforce.com and login.salesforce.com for Salesforce alerts

Note

  • Access to Salesforce is required from any Container Cloud cluster type.

  • If any additional Alertmanager notification receiver is enabled, for example, Slack, its endpoint must also be accessible from the cluster.

Caution

Regional clusters are unsupported since Container Cloud 2.25.0. Mirantis does not perform functional integration testing of the feature and intends to remove the related code in Container Cloud 2.26.0. If you still require this feature, contact Mirantis support for further information.

Reference hardware configuration

The following hardware configuration is used as a reference to deploy Mirantis Container Cloud with bare metal Container Cloud clusters with Mirantis Kubernetes Engine.

Reference hardware configuration for Container Cloud management and managed clusters on bare metal

Server role

Management cluster

Managed cluster

# of servers 0

3 1

6 2

CPU sockets

Minimal: 1
Recommended: 2
Minimal: 1
Recommended: 2

RAM, GB

Minimal: 64
Recommended: 128
Minimal: 64
Recommended: 256

System disk, GB 3

Minimal: SSD 1x 128
Recommended: NVME 1 x 960
Minimal: SSD 1 x 128
Recommended: NVME 1 x 960

SSD/HDD storage, GB

1x 1900 4

2x 1900

Onboard LAN ports

2

2

Discrete NICs

2

2

Total LAN ports 5

6

6

0

The Container Cloud reference architecture uses the following hardware models:

  • Server - Supermicro 1U SYS-6018R-TDW

  • CPU - Intel Xeon E5-2620v4

  • Discrete NICs - Intel X520-DA2

1

Adding more than 3 nodes to a management or regional cluster is not supported.

2

Three manager nodes for HA and three worker storage nodes for a minimal Ceph cluster.

3

A management cluster requires 2 volumes for Container Cloud (total 50 GB) and 5 volumes for StackLight (total 60 GB). A managed cluster requires 5 volumes for StackLight.

4

In total, at least 2 disks are required:

  • disk0 - minimum 120 GB for system

  • disk1 - minimum 120 GB for LocalVolumeProvisioner

For the default storage schema, see Default configuration of the host system storage

5

Only one PXE port per node is allowed. OOB management (IPMI) port is not included.

System requirements for the seed node

The seed node is necessary only to deploy the management cluster. When the bootstrap is complete, the bootstrap node can be redeployed and its resources can be reused for the managed cluster workloads.

The minimum reference system requirements for a baremetal-based bootstrap seed node are as follows:

  • Basic server on Ubuntu 20.04 with the following configuration:

    • Kernel version 4.15.0-76.86 or later

    • 8 GB of RAM

    • 4 CPU

    • 10 GB of free disk space for the bootstrap cluster cache

  • No DHCP or TFTP servers on any NIC networks

  • Routable access IPMI network for the hardware servers. For more details, see Host networking.

  • Internet access for downloading of all required artifacts

Network fabric

The following diagram illustrates the physical and virtual L2 underlay networking schema for the final state of the Mirantis Container Cloud bare metal deployment.

_images/bm-cluster-physical-and-l2-networking.png

The network fabric reference configuration is a spine/leaf with 2 leaf ToR switches and one out-of-band (OOB) switch per rack.

Reference configuration uses the following switches for ToR and OOB:

  • Cisco WS-C3560E-24TD has 24 of 1 GbE ports. Used in OOB network segment.

  • Dell Force 10 S4810P has 48 of 1/10GbE ports. Used as ToR in Common/PXE network segment.

In the reference configuration, all odd interfaces from NIC0 are connected to TOR Switch 1, and all even interfaces from NIC0 are connected to TOR Switch 2. The Baseboard Management Controller (BMC) interfaces of the servers are connected to OOB Switch 1.

The following recommendations apply to all types of nodes:

  • Use the Link Aggregation Control Protocol (LACP) bonding mode with MC-LAG domains configured on leaf switches. This corresponds to the 802.3ad bond mode on hosts.

  • Use ports from different multi-port NICs when creating bonds. This makes network connections redundant if failure of a single NIC occurs.

  • Configure the ports that connect servers to the PXE network with PXE VLAN as native or untagged. On these ports, configure LACP fallback to ensure that the servers can reach DHCP server and boot over network.

DHCP range requirements for PXE

When setting up the network range for DHCP Preboot Execution Environment (PXE), keep in mind several considerations to ensure smooth server provisioning:

  • Determine the network size. For instance, if you target a concurrent provision of 50+ servers, a /24 network is recommended. This specific size is crucial as it provides sufficient scope for the DHCP server to provide unique IP addresses to each new Media Access Control (MAC) address, thereby minimizing the risk of collision.

    The concept of collision refers to the likelihood of two or more devices being assigned the same IP address. With a /24 network, the collision probability using the SDBM hash function, which is used by the DHCP server, is low. If a collision occurs, the DHCP server provides a free address using a linear lookup strategy.

  • In the context of PXE provisioning, technically, the IP address does not need to be consistent for every new DHCP request associated with the same MAC address. However, maintaining the same IP address can enhance user experience, making the /24 network size more of a recommendation than an absolute requirement.

  • For a minimal network size, it is sufficient to cover the number of concurrently provisioned servers plus one additional address (50 + 1). This calculation applies after covering any exclusions that exist in the range. You can define excludes in the corresponding field of the Subnet object. For details, see API Reference: Subnet resource.

  • When the available address space is less than the minimum described above, you will not be able to automatically provision all servers. However, you can manually provision them by combining manual IP assignment for each bare metal host with manual pauses. For these operations, use the host.dnsmasqs.metal3.io/address and baremetalhost.metal3.io/detached annotations in the BareMetalHost object. For details, see API Reference: BareMetalHost resource.

  • All addresses within the specified range must remain unused before provisioning. If an IP address in-use is issued by the DHCP server to a BOOTP client, that specific client cannot complete provisioning.

Management cluster storage

The management cluster requires minimum two storage devices per node. Each device is used for different type of storage.

  • The first device is always used for boot partitions and the root file system. SSD is recommended. RAID device is not supported.

  • One storage device per server is reserved for local persistent volumes. These volumes are served by the Local Storage Static Provisioner (local-volume-provisioner) and used by many services of Container Cloud.

You can configure host storage devices using the BareMetalHostProfile resources. For details, see Customize the default bare metal host profile.

Requirements for an OpenStack-based cluster

While planning the deployment of an OpenStack-based Mirantis Container Cloud cluster with Mirantis Kubernetes Engine (MKE), consider the following general requirements:

  • Kubernetes on OpenStack requires the Cinder API V3 and Octavia API availability.

  • Mirantis supports deployments based on OpenStack Victoria or Yoga with Open vSwitch (OVS) or Tungsten Fabric (TF) on top of Mirantis OpenStack for Kubernetes (MOSK) Victoria or Yoga with TF.

For system requirements for a bootstrap node, see Requirements for a bootstrap node.

If you use a firewall or proxy, make sure that the bootstrap, management, and regional clusters have access to the following IP ranges and domain names required for the Container Cloud content delivery network and alerting:

  • IP ranges:

  • Domain names:

    • mirror.mirantis.com and repos.mirantis.com for packages

    • binary.mirantis.com for binaries and Helm charts

    • mirantis.azurecr.io and *.blob.core.windows.net for Docker images

    • mcc-metrics-prod-ns.servicebus.windows.net:9093 for Telemetry (port 443 if proxy is enabled)

    • mirantis.my.salesforce.com and login.salesforce.com for Salesforce alerts

Note

  • Access to Salesforce is required from any Container Cloud cluster type.

  • If any additional Alertmanager notification receiver is enabled, for example, Slack, its endpoint must also be accessible from the cluster.

Caution

Regional clusters are unsupported since Container Cloud 2.25.0. Mirantis does not perform functional integration testing of the feature and intends to remove the related code in Container Cloud 2.26.0. If you still require this feature, contact Mirantis support for further information.

Note

The requirements in this section apply to the latest supported Container Cloud release.

Requirements for an OpenStack-based Container Cloud cluster

Resource

Management or regional cluster

Managed cluster

Comments

# of nodes

3 (HA) + 1 (Bastion)

5 (6 with StackLight HA)

  • A bootstrap cluster requires access to the OpenStack API.

  • Each management or regional cluster requires 3 nodes for the manager nodes HA. Adding more than 3 nodes to a management or regional cluster is not supported.

  • A managed cluster requires 3 manager nodes for HA and 2 worker nodes for the Container Cloud workloads. If the multiserver mode is enabled for StackLight, 3 worker nodes are required for workloads.

  • Each management or regional cluster requires 1 node for the Bastion instance that is created with a public IP address to allow SSH access to instances.

# of vCPUs per node

8

8

  • The Bastion node requires 1 vCPU.

  • Refer to the RAM recommendations described below to plan resources for different types of nodes.

RAM in GB per node

24

16

To prevent issues with low RAM, Mirantis recommends the following types of instances for a managed cluster with 50-200 nodes:

  • 16 vCPUs and 32 GB of RAM - manager node

  • 16 vCPUs and 128 GB of RAM - nodes where the StackLight server components run

The Bastion node requires 1 GB of RAM.

Storage in GB per node

120

120

  • For the Bastion node, the default amount of storage is enough

  • To boot machines from a block storage volume, verify that disks performance matches the etcd requirements as described in etcd documentation

  • To boot the Bastion node from a block storage volume, 80 GB is enough

Operating system

Ubuntu 20.04
CentOS 7.9 0
Ubuntu 20.04
CentOS 7.9 0

For management, regional, and managed clusters, a base Ubuntu 20.04 or CentOS 7.9 image must be present in Glance.

MCR

23.0.7 Since 16.0.0
20.10.17 Since 14.0.0
20.10.13 Before 14.0.0
23.0.7 Since 16.0.0
20.10.17 Since 14.0.0
20.10.13 Before 14.0.0

Mirantis Container Runtime (MCR) is deployed by Container Cloud as a Container Runtime Interface (CRI) instead of Docker Engine.

OpenStack version

Queens, Victoria, Yoga

Queens, Victoria, Yoga

OpenStack Victoria and Yoga are supported on top of MOSK clusters.

Obligatory OpenStack components

Octavia, Cinder, OVS/TF

Octavia, Cinder, OVS/TF

  • Tungsten Fabric is supported on OpenStack Victoria or Yoga.

  • Only Cinder API V3 is supported.

# of Cinder volumes

7 (total 110 GB)

5 (total 60 GB)

  • Each management or regional cluster requires 2 volumes for Container Cloud (total 50 GB) and 5 volumes for StackLight (total 60 GB)

  • A managed cluster requires 5 volumes for StackLight

# of load balancers

10 (management) + 7 (regional)

6

  • LBs for a management cluster:

    • 1 for MKE

    • 1 for Container Cloud UI

    • 1 for Keycloak service

    • 1 for IAM service

    • 6 for StackLight

  • LBs for a regional cluster:

    • 1 for MKE

    • 6 for StackLight

  • LBs for a managed cluster:

    • 1 for MKE

    • 5 for StackLight with enabled logging (or 4 without logging)

# of floating IPs

11 (management) + 8 (regional)

11

  • FIPs for a management cluster:

    • 1 for MKE

    • 1 for Container Cloud UI

    • 1 for Keycloak service

    • 1 for IAM service

    • 1 for the Bastion node (or 3 without Bastion: one FIP per manager node)

    • 6 for StackLight

  • FIPs for a regional cluster:

    • 1 for MKE

    • 1 for the Bastion node (or 3 without Bastion)

    • 6 for StackLight

  • FIPs for a managed cluster:

    • 1 for MKE

    • 3 for the manager nodes

    • 2 for the worker nodes

    • 5 for StackLight with enabled logging (4 without logging)

0(1,2)

A Container Cloud cluster based on both Ubuntu and CentOS operating systems is not supported.

Requirements for a VMware vSphere-based cluster

Note

Container Cloud is developed and tested on VMware vSphere 7.0 and 6.7.

For system requirements for a bootstrap node, see Requirements for a bootstrap node.

If you use a firewall or proxy, make sure that the bootstrap, management, and regional clusters have access to the following IP ranges and domain names required for the Container Cloud content delivery network and alerting:

  • IP ranges:

  • Domain names:

    • mirror.mirantis.com and repos.mirantis.com for packages

    • binary.mirantis.com for binaries and Helm charts

    • mirantis.azurecr.io and *.blob.core.windows.net for Docker images

    • mcc-metrics-prod-ns.servicebus.windows.net:9093 for Telemetry (port 443 if proxy is enabled)

    • mirantis.my.salesforce.com and login.salesforce.com for Salesforce alerts

Note

  • Access to Salesforce is required from any Container Cloud cluster type.

  • If any additional Alertmanager notification receiver is enabled, for example, Slack, its endpoint must also be accessible from the cluster.

Caution

Regional clusters are unsupported since Container Cloud 2.25.0. Mirantis does not perform functional integration testing of the feature and intends to remove the related code in Container Cloud 2.26.0. If you still require this feature, contact Mirantis support for further information.

Note

The requirements in this section apply to the latest supported Container Cloud release.

Requirements for a vSphere-based Container Cloud cluster

Resource

Management cluster

Managed cluster

Comments

# of nodes

3 (HA)

5 (6 with StackLight HA)

  • A bootstrap cluster requires access to the vSphere API.

  • A management cluster requires 3 nodes for the manager nodes HA. Adding more than 3 nodes to a management or regional cluster is not supported.

  • A managed cluster requires 3 manager nodes for HA and 2 worker nodes for the Container Cloud workloads. If the multiserver mode is enabled for StackLight, 3 worker nodes are required for workloads.

# of vCPUs per node

8

8

Refer to the RAM recommendations described below to plan resources for different types of nodes.

RAM in GB per node

32

16

To prevent issues with low RAM, Mirantis recommends the following VM templates for a managed cluster with 50-200 nodes:

  • 16 vCPUs and 40 GB of RAM - manager node

  • 16 vCPUs and 128 GB of RAM - nodes where the StackLight server components run

Storage in GB per node

120

120

The listed amount of disk space must be available as a shared datastore of any type, for example, NFS or vSAN, mounted on all hosts of the vCenter cluster.

Operating system

RHEL 8.7 or 7.9 1
CentOS 7.9 1
Ubuntu 20.04
RHEL 8.7 or 7.9 1
CentOS 7.9 1
Ubuntu 20.04

For a management and managed cluster, a base OS VM template must be present in the VMware VM templates folder available to Container Cloud. For details about the template, see Prepare the virtual machine template.

RHEL license
(for RHEL deployments only)

RHEL licenses for Virtual Datacenters

RHEL licenses for Virtual Datacenters

This license type allows running unlimited guests inside one hypervisor. The amount of licenses is equal to the amount of hypervisors in vCenter Server, which will be used to host RHEL-based machines. Container Cloud will schedule machines according to scheduling rules applied to vCenter Server. Therefore, make sure that your RedHat Customer portal account has enough licenses for allowed hypervisors.

MCR

23.0.7 Since 14.1.0
20.10.17 Since 14.0.0
23.0.7 Since 14.1.0
20.10.17 Since 14.0.0

Mirantis Container Runtime (MCR) is deployed by Container Cloud as a Container Runtime Interface (CRI) instead of Docker Engine.

VMware vSphere version

7.0, 6.7

7.0, 6.7

cloud-init version

19.4 for RHEL/CentOS 7.9
20.3 for RHEL 8.7 TechPreview
19.4 for RHEL/CentOS 7.9
20.3 for RHEL 8.7 TechPreview

The minimal cloud-init package version built for the Prepare the virtual machine template.

VMware Tools version

11.0.5

11.0.5

The minimal open-vm-tools package version built for the Prepare the virtual machine template.

Obligatory vSphere capabilities

DRS,
Shared datastore
DRS,
Shared datastore

A shared datastore must be mounted on all hosts of the vCenter cluster. Combined with Distributed Resources Scheduler (DRS), it ensures that the VMs are dynamically scheduled to the cluster hosts.

IP subnet size

/24

/24

Consider the supported VMware vSphere network objects and IPAM recommendations.

Minimal IP addresses distribution:

  • Management cluster:

    • 1 for the load balancer of Kubernetes API

    • 3 for manager nodes (one per node)

    • 6 for the Container Cloud services

    • 6 for StackLight

  • Managed cluster:

    • 1 for the load balancer of Kubernetes API

    • 3 for manager nodes

    • 2 for worker nodes

    • 6 for StackLight

1(1,2,3,4)
  • CentOS 7.9 is available as Technology Preview. Use this configuration for testing and evaluation purposes only.

  • RHEL 8.7 is generally available since Cluster releases 16.0.0 and 14.1.0. Before these Cluster releases, it is supported as Technology Preview.

  • A Container Cloud cluster based on mixed operating systems, such as RHEL and CentOS, or on mixed versions of RHEL, such as RHEL 7.9 and 8.7, is not supported.

StackLight requirements for an MKE attached cluster

Available since 2.25.2

During attachment of a Mirantis Kubernetes Engine (MKE) cluster that is not deployed by Container Cloud to a vSphere-based management cluster, you can add StackLight as the logging, monitoring, and alerting solution. In this scenario, your cluster must satisfy several requirements that primarily involve alignment of cluster resources with specific StackLight settings.

General requirements

While planning the attachment of an existing MKE cluster that is not deployed by Container Cloud to a vSphere-based management cluster, consider the following general requirements for StackLight:

Note

Attachment of MKE clusters is tested on Ubuntu 20.04.

Requirements for cluster size

While planning the attachment of an existing MKE cluster that is not deployed by Container Cloud to a vSphere-based management cluster, consider the cluster size requirements for StackLight. Depending on the following specific StackLight HA and logging settings, use the example size guidelines below:

  • The non-HA mode - StackLight services are installed on a minimum of one node with the StackLight label (StackLight nodes) with no redundancy using Persistent Volumes (PVs) from the default storage class to store data. Metric collection agents are installed on each node (Other nodes).

  • The HA mode - StackLight services are installed on a minimum of three nodes with the StackLight label (StackLight nodes) with redundancy using PVs provided by Local Volume Provisioner to store data. Metric collection agents are installed on each node (Other nodes).

  • Logging enabled - the Enable logging option is turned on, which enables the OpenSearch cluster to store infrastructure logs.

  • Logging disabled - the Enable logging option is turned off. In this case, StackLight will not install OpenSearch and will not collect infrastructure logs.

LoadBalancer (LB) Services support is required to provide external access to StackLight web UIs.

StackLight requirements for an attached MKE cluster, with logging enabled:

StackLight nodes 1

Other nodes

Storage (PVs)

LBs

Non-HA (1-node example)

  • RAM requests: 11 GB

  • RAM limits: 33 GB

  • CPU requests: 4.5 cores

  • CPU limits: 12 cores

  • RAM requests: 0.25 GB

  • RAM limits: 1 GB

  • CPU requests: 0.5 cores

  • CPU limits: 1 core

  • 1 PV for Prometheus (size is configurable; 1x total)

  • 2 PVs for Alertmanager (2 Gi/volume; 4 Gi total)

  • 1 PV for Patroni (10 G; 10 G total)

  • 1 PV for OpenSearch (size is configurable; 1x total)

5

HA (3-nodes example)

  • RAM requests: 10 GB

  • RAM limits: 25 GB

  • CPU requests: 2.8 cores

  • CPU limits: 7.5 cores

  • RAM requests: 0.25 GB

  • RAM limits: 1 GB

  • CPU requests: 0.5 cores

  • CPU limits: 1 core

  • 2 PVs (1 per StackLight node) for Prometheus (size is configurable; 2x total)

  • 2 PVs (1 per StackLight node) for Alertmanager (2 Gi/volume; 4 Gi total)

  • 3 PVs (1 per StackLight node) for Patroni (10 G/volume; 30 G total)

  • 3 PVs (1 per StackLight node) for OpenSearch (size is configurable; 3x total)

5

StackLight requirements for an attached MKE cluster, with logging disabled

StackLight nodes 1

Other nodes

Storage (PVs)

LBs

Non-HA (1-node example)

  • RAM requests: 4 GB

  • RAM limits: 23 GB

  • CPU requests: 3 cores

  • CPU limits: 9 cores

  • RAM requests: 0.05 GB

  • RAM limits: 0.1 GB

  • CPU requests: 0.01 cores

  • CPU limits: 0 cores

  • 1 PV for Prometheus (size is configurable; 1x total)

  • 2 PVs for Alertmanager (2 Gi/volume; 4Gi total)

  • 1 PV for Patroni (10 G; 10 G total)

4

HA (3-nodes example)

  • RAM requests: 3 GB

  • RAM limits: 15 GB

  • CPU requests: 1.6 cores

  • CPU limits: 4.2 cores

  • RAM requests: 0.05 GB

  • RAM limits: 0.1 GB

  • CPU requests: 0.01 cores

  • CPU limits: 0 core

  • 2 PVs (1 per StackLight node) for Prometheus (size is configurable; 2x total)

  • 2 PVs (1 per StackLight node) for Alertmanager (2 Gi/volume; 4 Gi total)

  • 3 PVs (1 per StackLight node) for Patroni (10 G/volume; 30 G total)

4

1(1,2)

In the non-HA mode, StackLight components are bound to the nodes labeled with the StackLight label. If there are no nodes labeled, StackLight components will be scheduled to all schedulable worker nodes until the StackLight label(s) are added. The requirements presented in the table for the non-HA mode are summarized requirements for all StackLight nodes.

Proxy and cache support

Proxy support

If you require all Internet access to go through a proxy server for security and audit purposes, you can bootstrap management and regional clusters using proxy. The proxy server settings consist of three standard environment variables that are set prior to the bootstrap process:

  • HTTP_PROXY

  • HTTPS_PROXY

  • NO_PROXY

These settings are not propagated to managed clusters. However, you can enable a separate proxy access on a managed cluster using the Container Cloud web UI. This proxy is intended for the end user needs and is not used for a managed cluster deployment or for access to the Mirantis resources.

Caution

Since Container Cloud uses the OpenID Connect (OIDC) protocol for IAM authentication, management clusters require a direct non-proxy access from regional and managed clusters.

StackLight components, which require external access, automatically use the same proxy that is configured for Container Cloud clusters.

On the managed clusters with limited Internet access, a proxy is required for StackLight components that use HTTP and HTTPS and are disabled by default but need external access if enabled, for example, for the Salesforce integration and Alertmanager notifications external rules. For more details about proxy implementation in StackLight, see StackLight proxy.

For the list of Mirantis resources and IP addresses to be accessible from the Container Cloud clusters, see Hardware and system requirements.

After enabling proxy support on regional and managed clusters, proxy is used for:

  • Docker and CDN traffic on regional clusters

  • Docker traffic on managed clusters

  • StackLight

  • OpenStack on MOSK-based clusters

Artifacts caching

The Container Cloud managed clusters are deployed without direct Internet access in order to consume less Internet traffic in your cloud. The Mirantis artifacts used during managed clusters deployment are downloaded through a cache running on a regional cluster. The feature is enabled by default on new managed clusters and will be automatically enabled on existing clusters during upgrade to the latest version.

Caution

IAM operations require a direct non-proxy access of a managed cluster to a management cluster.

Firewall configuration

This section includes the details about ports and protocols used in a Container Cloud deployment.

Container Cloud
Mirantis Container Cloud – LCM

Component

Network

Protocol

Port

Consumers

Web UI, cache, Kubernetes API, and others

LCM API/Mgmt

TCP

443, 6443

External clients

Squid Proxy

LCM API/Mgmt

TCP

3128

Applicable to the vSphere provider only. All nodes in management and managed clusters.

SSH

LCM API/Mgmt

TCP

22

External clients

Chrony

LCM_API/Mgmt

TCP

323

All nodes in management and managed clusters.

NTP

LCM_API/Mgmt

UDP

123

All nodes in management and managed clusters.

LDAP

LCM API/Mgmt

UDP

389

LDAPs

LCM API/Mgmt

TCP/UDP

686

Mirantis Container Cloud – Bare metal

Component

Network

Protocol

Port

Ironic

LCM 0

TCP/UDP

  • TCP: 9999, 6385, 8089, 5050, 9797, 601

  • UDP: 9999, 514

Ironic syslog

PXE

TCP/UDP

  • TCP: 601

  • UDP: 514

Ironic image repo

PXE

TCP

80

MKE/Kubernetes API

LCM 0

TCP/UDP

  • TCP: 179, 2376, 2377, 7946, 10250, 12376, 12379-12388

  • UDP: 4789, 7946

BOOTP

PXE

UDP

68

DHCP server

PXE

UDP

67

IPMI

PXE/LCM 0

TCP/UDP

  • TCP: 623 1

  • UDP: 623

SSH

PXE/LCM

TCP

22

DNS

LCM 0

TCP/UDP

53

NTP

LCM 0

TCP/UDP

123

TFTP

PXE

UDP

69

Squid Proxy

LCM 0

TCP

3128

LDAP

LCM 0

TCP

636

HTTPS

LCM 0

TCP

443

StackLight

LCM 0

TCP

  • 9091

  • 9126

  • 19100 Since 17.0.0, 16.0.0, 14.1.0

  • 9100 Before 17.0.0, 16.0.0, 14.1.0

0(1,2,3,4,5,6,7,8,9)

Depends on the default route.

1

Depends on the Baseboard Management Controller (BMC) protocol, defaults to IPMI.

Mirantis Kubernetes Engine

For available Mirantis Kubernetes Engine (MKE) ports, refer to MKE Documentation: Open ports to incoming traffic.

StackLight

The tables below contain the details about ports and protocols used by different StackLight components.

Warning

This section does not describe communications within the cluster network.

User interfaces

Component

Network

Direction

Port/Protocol

Consumer

Comments

Alerta UI

External network (LB service)

Inbound

443/TCP/HTTPS

Cluster users

Add the assigned external IP to the allowlist.

Alertmanager UI

External network (LB service)

Inbound

443/TCP/HTTPS

Cluster users

Add the assigned external IP to the allowlist.

Grafana UI

External network (LB service)

Inbound

443/TCP/HTTPS

Cluster users

Add the assigned external IP to the allowlist.

OpenSearch Dashboards UI

External network (LB service)

Inbound

443/TCP/HTTPS

Cluster users

Only when the StackLight logging stack is enabled. Add the assigned external IP to the allowlist.

Prometheus UI

External network (LB service)

Inbound

443/TCP/HTTPS

Cluster users

Add the assigned external IP to the allowlist.

Alertmanager notifications receivers

Component

Network

Direction

Port/Protocol

Destination

Comments

Alertmanager Email notifications integration

Cluster network

Outbound

TCP/SMTP

Depends on the configuration, see the comment.

Only when email notifications are enabled. Add an SMTP host URL to the allowlist.

Alertmanager Microsoft Teams notifications integration

Cluster network

Outbound

TCP/HTTPS

Depends on the configuration, see the comment.

Only when Microsoft Teams notifications are enabled. Add a webhook URL to the allowlist.

Alertmanager Salesforce notifications integration

Cluster network

Outbound

TCP/HTTPS

For Mirantis support mirantis.my.salesforce.com and login.salesforce.com. Depends on the configuration, see the comment.

Only when Salesforce notifications are enabled. Add an SF instance URL and an SF login URL to the allowlist. See Requirements for a baremetal-based cluster for details.

Alertmanager ServiceNow notifications integration

Cluster network

Outbound

TCP/HTTPS

Depends on the configuration, see the comment.

Only when notifications to ServiceNow are enabled. Add a configured ServiceNow URL to the allowlist.

Alertmanager Slack notifications integration

Cluster network

Outbound

TCP/HTTPS

Depends on the configuration, see the comment.

Only when notifications to Slack are enabled. Add a configured Slack URL to the allowlist.

Notification integration of Alertmanager generic receivers

Cluster network

Outbound

Customizable, see the comment

Depends on the configuration, see the comment.

Only when any custom Alertmanager integration is enabled. Depending on the integration type, add the corresponding URL to the allowlist.

External integrations

Component

Network

Direction

Port/Protocol

Destination

Comments

Salesforce reporter

Cluster network

Outbound

TCP/HTTPS

For Mirantis support mirantis.my.salesforce.com and login.salesforce.com. Depends on the configuration, see the comment.

Only when the Salesforce reporter is enabled. Add a SF instance URL and SF login URL to the allowlist. See Requirements for a baremetal-based cluster for details.

Prometheus Remote Write

Cluster network

Outbound

TCP

Depends on the configuration, see the comment.

Only when the Prometheus Remote Write feature is enabled. Add a configured remote write destination URL to the allowlist.

Prometheus custom scrapes

Cluster network

Outbound

TCP

Depends on the configuration, see the comment.

Only when the Custom Prometheus scrapes feature is enabled. Add configured scrape targets to the allowlist.

Fluentd remote syslog output

Cluster network

Outbound

TCP or UDP (protocol and port are configurable)

Depends on the configuration, see the comment.

Only when the Logging to remote Syslog feature is enabled. Add a configured remote syslog URL to the allowlist.

Metric Collector

Cluster network

Outbound

9093/443/TCP

mcc-metrics-prod-ns.servicebus.windows.net

Applicable to management clusters only. Add a specific URL from Microsoft Azure to the allowlist. See Requirements for a baremetal-based cluster for details.

External Endpoint monitoring

Cluster network

Outbound

TCP/HTTP(S)

Depends on the configuration, see the comment.

Only when the External endpoint monitoring feature is enabled. Add configured monitored URLs to the allowlist.

SSL certificate monitoring

Cluster network

Outbound

TCP/HTTP(S)

Depends on the configuration, see the comment.

Only when SSL certificates monitoring feature is enabled. Add configured monitored URLs to the allowlist.

Metrics exporters

Component

Network

Direction

Port/Protocol

Consumer

Comments

Prometheus Node Exporter

Host network

Inbound (from cluster network)

19100/TCP Since 17.0.0, 16.0.0, 14.1.0, 9100/TCP Before 17.0.0, 16.0.0, 14.1.0

Prometheus from the stacklight namespace

Prometheus from Cluster network scrape metrics from all nodes.

Fluentd (Prometheus metrics endpoint)

Host network

Inbound (from cluster network)

24231/TCP

Prometheus from the stacklight namespace

Only when the StackLight logging stack is enabled. Prometheus from the cluster network scrapes metrics from all nodes.

Calico node

Host network

Inbound (from cluster network)

9091/TCP

Prometheus from the stacklight namespace

Prometheus from cluster network scrape metrics from all nodes.

Telegraf SMART plugin

Host network

Inbound (from cluster network)

9126/TCP

Prometheus from the stacklight namespace

Applicable to the bare metal provider obly. Prometheus from scrapes metrics of the cluster network from all nodes.

MKE Manager API

Host network

Inbound (from cluster network)

4443/TCP, 6443/TCP

Blackbox exporter from the stacklight namespace

Applicable to the master node only. Blackbox exporter from cluster network probes all master nodes.

  • 6443/TCP is applicable to the OpenStack provider only.

  • 4443/TCP is applicable to the bare metal and vSphere providers only.

On attached MKE clusters, the port and protocol depend on the MKE cluster configuration.

MKE Metrics Engine

Host network

Inbound (from cluster network)

12376/TCP

Prometheus from the stacklight namespace

Prometheus from cluster network scrape metrics from all nodes.

Kubernetes Master API

Host network

Inbound (from cluster network)

443/TCP, 5443/TCP

Blackbox exporter from the stacklight namespace

Applicable to the master node only. Blackbox exporter from cluster network probes all master nodes.

  • 443/TCP is applicable to the OpenStack provider only and to attached MKE clusters.

  • 5443/TCP is applicable to the bare metal and vSphere providers only.

Container Cloud telemetry

Component

Network

Direction

Port/Protocol

Consumer

Destination

Comments

Telemeter client

Cluster network (managed cluster)

Outbound (to regional cluster external LB)

443/TCP

n/a

Telemeter server on a regional cluster (Telemeter server external IP from the stacklight namespace of a regional cluster)

Applicable to managed clusters only. The Telemeter client on a managed cluster pushes metrics to the Telemeter server on a regional cluster.

Cluster network (regional cluster)

Outbound (to management cluster external LB)

443/TCP

n/a

Telemeter server on a management cluster (Telemeter server external IP from the stacklight namespace of a management cluster)

Applicable to regional clusters only. The Telemeter client on a regional cluster pushes metrics to the Telemeter server on a management cluster.

Telemeter server

External network (LB service)

Inbound (from regional cluster network)

443/TCP

Telemeter client on regional clusters

n/a

Applicable to management clusters only. The Telemeter client on the regional cluster pushes metrics to the Telemeter server on the management cluster.

External network (LB service)

Inbound (from managed cluster network)

443/TCP

Telemeter client on managed clusters

n/a

Applicable to regional clusters only. The Telemeter client on a managed cluster pushes metrics to the Telemeter server on the regional cluster.

Ceph

Ceph monitors use their node host networks to interact with Ceph daemons. Ceph daemons communicate with each other over a specified cluster network and provide endpoints over the public network.

The messenger V2 (msgr2) or earlier V1 (msgr) protocols are used for communication between Ceph daemons.

Ceph daemon

Network

Protocol

Port

Description

Consumers

Manager (mgr)

Cluster network

msgr/msgr2

6800

Listens on the first available port of the 6800-7300 range

csi-rbdplugin,
csi-rbdprovisioner,
rook-ceph-mon

Metadata server (mds)

Cluster network

msgr/msgr2

6800

Listens on the first available port of the 6800-7300 range

csi-cephfsplugin,
csi-cephfsprovisioner

Monitor (mon)

LCM host network

msgr/msgr2

msgr:3300,
msgr2:6789

Monitor has separate ports for msgr and msgr2

Ceph clients
rook-ceph-osd,
rook-ceph-rgw

Ceph OSD (osd)

Cluster network

msgr/msgr2

6800-7300

Binds to the first available port from the 6800-7300 range

rook-ceph-mon,
rook-ceph-mgr,
rook-ceph-mds

Mirantis Kubernetes Engine API limitations

To ensure the Mirantis Container Cloud stability in managing the Container Cloud-based Mirantis Kubernetes Engine (MKE) clusters, the following MKE API functionality is not available for the Container Cloud-based MKE clusters as compared to the MKE clusters that are deployed not by Container Cloud. Use the Container Cloud web UI or CLI for this functionality instead.

Public APIs limitations in a Container Cloud-based MKE cluster

API endpoint

Limitation

GET /swarm

Swarm Join Tokens are filtered out for all users, including admins.

PUT /api/ucp/config-toml

All requests are forbidden.

POST /nodes/{id}/update

Requests for the following changes are forbidden:

  • Change Role

  • Add or remove the com.docker.ucp.orchestrator.swarm and com.docker.ucp.orchestrator.kubernetes labels.

DELETE /nodes/{id}

All requests are forbidden.

Since 2.25.1, Container Cloud does not override changes in MKE configuration except the following list of parameters that are automatically managed by Container Cloud and are always overridden when modified using the MKE API. However, you can manually configure a few options from this list using the Cluster object of a Container Cloud cluster. They are labeled with the superscript in the table. For details, see the Comments column.

MKE options managed by Container Cloud

Configuration option name

Option parameters

Comments

audit_log_configuration

  • level

  • support_dump_include_audit_logs

auth

  • backend

  • default_new_user_role

  • samlEnabled

auth.external_identity_provider

All

hardening_configuration

  • hardening_enabled

  • limit_kernel_capabilities

  • pids_limit_int

  • pids_limit_k8s

  • pids_limit_swarm

scheduling_configuration

All

tracking_configuration

cluster_label

cluster_config

  • calico_ip_auto_method

  • calico_mtu

  • calico_vxlan_mtu

  • calico_vxlan_port

  • calico_vxlan

  • cloud_provider

  • controller_port

  • custom_kube_api_server_flags

  • custom_kube_controller_manager_flags

  • custom_kube_scheduler_flags

  • custom_kubelet_flags

  • etcd_storage_quota

  • exclude_server_identity_headers

  • ipip_mtu

  • kube_api_server_auditing

  • kube_api_server_profiling_enabled

  • kube_apiserver_port

  • kube_protect_kernel_defaults

  • local_volume_collection_mapping

  • manager_kube_reserved_resources

  • metrics_retention_time

  • metrics_scrape_interval

  • nodeport_range

  • pod_cidr

  • priv_attributes_allowed_for_service_accounts

  • priv_attributes_service_accounts

  • profiling_enabled

  • prometheus_memory_limit

  • prometheus_memory_request

  • secure_overlay

  • service_cluster_ip_range

  • swarm_port

  • swarm_strategy

  • unmanaged_cni

  • vxlan_vni

  • worker_kube_reserved_resources

  • For calico_mtu, use the spec:providerSpec:value:calico:mtu parameter in the Cluster object. For details, see Set the MTU size for Calico.

  • For etcd_storage_quota, use the spec:providerSpec:value:etcd:storageQuota parameter in the Cluster object. For details, see Increase storage quota for etcd.

  • For priv_attributes parameters, you can add custom options on top of existing parameters using the MKE API.

Deployment Guide

Deploy a baremetal-based management cluster Deprecated

Caution

This bootstrap procedure is deprecated since Container Cloud 2.25.0. Instead, use Bootstrap v2 that provides best user experience to set up Container Cloud. For details, see Deploy Container Cloud using Boostrap v2.

This section describes how to bootstrap a baremetal-based Mirantis Container Cloud management cluster using Bootstrap v1.

Workflow overview

The bare metal management system enables the Infrastructure Operator to deploy Mirantis Container Cloud on a set of bare metal servers. It also enables Container Cloud to deploy managed clusters on bare metal servers without a pre-provisioned operating system.

The Infrastructure Operator performs the following steps to install Container Cloud in a bare metal environment:

  1. Install and connect hardware servers as described in Requirements for a baremetal-based cluster.

    Caution

    The baremetal-based Container Cloud does not manage the underlay networking fabric but requires specific network configuration to operate.

  2. Install Ubuntu 20.04 on one of the bare metal machines to create a seed node and copy the bootstrap tarball to this node.

  3. Obtain the Mirantis license file that will be required during the bootstrap.

  4. Create the deployment configuration files that include the bare metal hosts metadata.

  5. Validate the deployment templates using fast preflight.

  6. Run the bootstrap script for the fully automated installation of the management cluster onto the selected bare metal hosts.

Using the bootstrap script, the Container Cloud bare metal management system prepares the seed node for the management cluster and starts the deployment of Container Cloud itself. The bootstrap script performs all necessary operations to perform the automated management cluster setup. The deployment diagram below illustrates the bootstrap workflow of a baremetal-based management cluster.

_images/bm-bootstrap-workflow.png
Bootstrap a management cluster

This section describes how to prepare and bootstrap a baremetal-based management cluster. The procedure includes:

  • A runbook that describes how to create a seed node that is a temporary server used to run the management cluster bootstrap scripts.

  • A step-by-step instruction how to prepare metadata for the bootstrap scripts and how to run them.

Prepare the seed node

Before installing Mirantis Container Cloud on a bare metal environment, complete the following preparation steps:

  1. Verify that the hardware allocated for the installation meets the minimal requirements described in Requirements for a baremetal-based cluster.

  2. Install basic Ubuntu 20.04 server using standard installation images of the operating system on the bare metal seed node.

  3. Log in to the seed node that is running Ubuntu 20.04.

  4. Prepare the system and network configuration:

    1. Establish a virtual bridge using an IP address of the PXE network on the seed node. Use the following netplan-based configuration file as an example:

      # cat /etc/netplan/config.yaml
      network:
        version: 2
        renderer: networkd
        ethernets:
          ens3:
              dhcp4: false
              dhcp6: false
        bridges:
            br0:
                addresses:
                # Replace with IP address from PXE network to create a virtual bridge
                - 10.0.0.15/24
                dhcp4: false
                dhcp6: false
                # Adjust for your environment
                gateway4: 10.0.0.1
                interfaces:
                # Interface name may be different in your environment
                - ens3
                nameservers:
                    addresses:
                    # Adjust for your environment
                    - 8.8.8.8
                parameters:
                    forward-delay: 4
                    stp: false
      
    2. Apply the new network configuration using netplan:

      sudo netplan apply
      
    3. Verify the new network configuration:

      sudo apt update && sudo apt install -y bridge-utils
      sudo brctl show
      

      Example of system response:

      bridge name     bridge id               STP enabled     interfaces
      br0             8000.fa163e72f146       no              ens3
      

      Verify that the interface connected to the PXE network belongs to the previously configured bridge.

    4. Install the current Docker version available for Ubuntu 20.04:

      sudo apt-get update
      sudo apt-get install docker.io
      
    5. Verify that your logged USER has access to the Docker daemon:

      sudo usermod -aG docker $USER
      
    6. Log out and log in again to the seed node to apply the changes.

    7. Verify that Docker is configured correctly and has access to Container Cloud CDN. For example:

      docker run --rm alpine sh -c "apk add --no-cache curl; \
      curl https://binary.mirantis.com"
      

      The system output must contain a json file with no error messages. In case of errors, follow the steps provided in Troubleshooting.

      Note

      If you require all Internet access to go through a proxy server for security and audit purposes, configure Docker proxy settings as described in the official Docker documentation.

  5. Verify that the seed node has direct access to the Baseboard Management Controller (BMC) of each bare metal host. All target hardware nodes must be in the power off state.

    For example, using the IPMI tool:

    apt install ipmitool
    ipmitool -I lanplus -H 'IPMI IP' -U 'IPMI Login' -P 'IPMI password' \
    chassis power status
    

    Example of system response:

    Chassis Power is off
    
Configure BIOS on a bare metal host

Before adding new BareMetalHost objects, configure hardware hosts to correctly load them over the PXE network.

Important

Consider the following common requirements for hardware hosts configuration:

  • Update firmware for BIOS and Baseboard Management Controller (BMC) to the latest available version, especially if you are going to apply the UEFI configuration.

    Container Cloud uses the ipxe.efi binary loader that might be not compatible with old firmware and have vendor-related issues with UEFI booting. For example, the Supermicro issue. In this case, we recommend using the legacy booting format.

  • Configure all or at least the PXE NIC on switches.

    If the hardware host has more than one PXE NIC to boot, we strongly recommend setting up only one in the boot order. It speeds up the provisioning phase significantly.

    Some hardware vendors require a host to be rebooted during BIOS configuration changes from legacy to UEFI or vice versa for the extra option with NIC settings to appear in the menu.

  • Connect only one Ethernet port on a host to the PXE network at any given time. Collect the physical address (MAC) of this interface and use it to configure the BareMetalHost object describing the host.

To configure BIOS on a bare metal host:

  1. Enable the global BIOS mode using BIOS > Boot > boot mode select > legacy. Reboot the host if required.

  2. Enable the LAN-PXE-OPROM support using the following menus:

    • BIOS > Advanced > PCI/PCIe Configuration > LAB OPROM TYPE > legacy

    • BIOS > Advanced > PCI/PCIe Configuration > Network Stack > enabled

    • BIOS > Advanced > PCI/PCIe Configuration > IPv4 PXE Support > enabled

  3. Set up the configured boot order:

    1. BIOS > Boot > Legacy-Boot-Order#1 > Hard Disk

    2. BIOS > Boot > Legacy-Boot-Order#2 > NIC

  4. Save changes and power off the host.

  1. Enable the global BIOS mode using BIOS > Boot > boot mode select > UEFI. Reboot the host if required.

  2. Enable the LAN-PXE-OPROM support using the following menus:

    • BIOS > Advanced > PCI/PCIe Configuration > LAB OPROM TYPE > uefi

    • BIOS > Advanced > PCI/PCIe Configuration > Network Stack > enabled

    • BIOS > Advanced > PCI/PCIe Configuration > IPv4 PXE Support > enabled

    Note

    UEFI support might not apply to all NICs. But at least built-in network interfaces should support it.

  3. Set up the configured boot order:

    1. BIOS > Boot > UEFI-Boot-Order#1 > UEFI Hard Disk

    2. BIOS > Boot > UEFI-Boot-Order#1 > UEFI Network

  4. Save changes and power off the host.

Prepare metadata and deploy the management cluster since 2.24.0

This section describes how to prepare cluster metadata and deploy the management cluster since Container Cloud 2.24.0. For description of changes applied as compared to previous Container Cloud releases, see Release Notes: MetalLB configuration changes.

Using the example procedure below, replace the addresses and credentials in the configuration YAML files with the data corresponding to your environment. Keep everything else as is, including the file names and YAML structure.

To prepare metadata and deploy the management cluster:

  1. Log in to the seed node that you configured as described in Prepare the seed node.

  2. Change to your preferred work directory, for example:

    cd $HOME
    
  3. Prepare the bootstrap script:

    1. Download and run the Container Cloud bootstrap script:

      sudo apt-get update
      sudo apt-get install wget
      wget https://binary.mirantis.com/releases/get_container_cloud.sh
      chmod 0755 get_container_cloud.sh
      ./get_container_cloud.sh
      
    2. Change the directory to the kaas-bootstrap folder created by the script.

  4. Obtain your license file that will be required during the bootstrap:

    1. Select from the following options:

      • Open the email from support@mirantis.com with the subject Mirantis Container Cloud License File or Mirantis OpenStack License File

      • In the Mirantis CloudCare Portal, open the Account or Cloud page

    2. Download the License File and save it as mirantis.lic under the kaas-bootstrap directory on the bootstrap node.

    3. Verify that mirantis.lic contains the previously downloaded Container Cloud license by decoding the license JWT token, for example, using jwt.io.

      Example of a valid decoded Container Cloud license data with the mandatory license field:

      {
          "exp": 1652304773,
          "iat": 1636669973,
          "sub": "demo",
          "license": {
              "dev": false,
              "limits": {
                  "clusters": 10,
                  "workers_per_cluster": 10
              },
              "openstack": null
          }
      }
      

      Warning

      The MKE license does not apply to mirantis.lic. For details about MKE license, see MKE documentation.

  5. Prepare the deployment templates:

    Note

    The serviceusers.yaml.template and bootstrapregion.yaml.template files relate to the Bootstrap v2 procedure only. Therefore, skip these templates in the Bootstrap v1 deployments.

    For details on the Bootstrap v2 procedure, see Deploy Container Cloud using Boostrap v2.

    1. Create a copy of the current templates directory for future reference:

      mkdir templates.backup
      cp -r templates/*  templates.backup/
      
    2. Inspect the default bare metal host profile definition in templates/bm/baremetalhostprofiles.yaml.template and adjust it to fit your hardware configuration. For details, see Customize the default bare metal host profile.

      Warning

      All data will be wiped during cluster deployment on devices defined directly or indirectly in the fileSystems list of BareMetalHostProfile. For example:

      • A raw device partition with a file system on it

      • A device partition in a volume group with a logical volume that has a file system on it

      • An mdadm RAID device with a file system on it

      • An LVM RAID device with a file system on it

      The wipe field is always considered true for these devices. The false value is ignored.

      Therefore, to prevent data loss, move the necessary data from these file systems to another server beforehand, if required.

    3. In templates/bm/baremetalhosts.yaml.template, update the bare metal host definitions according to your environment configuration. Use the table below for reference.

      • Manually set all parameters that start with SET_.

      • Set the corresponding value for the kaas.mirantis.com/region label across all objects listed in templates/bm/baremetalhosts.yaml.template.

      Bare metal hosts template mandatory parameters

      Parameter

      Description

      Example value

      SET_MACHINE_0_IPMI_USERNAME

      The IPMI user name to access the BMC. 0

      user

      SET_MACHINE_0_IPMI_PASSWORD

      The IPMI password to access the BMC. 0

      password

      SET_MACHINE_0_MAC

      The MAC address of the first master node in the PXE network.

      ac:1f:6b:02:84:71

      SET_MACHINE_0_BMC_ADDRESS

      The IP address of the BMC endpoint for the first master node in the cluster. Must be an address from the OOB network that is accessible through the management network gateway.

      192.168.100.11

      SET_MACHINE_1_IPMI_USERNAME

      The IPMI user name to access the BMC. 0

      user

      SET_MACHINE_1_IPMI_PASSWORD

      The IPMI password to access the BMC. 0

      password

      SET_MACHINE_1_MAC

      The MAC address of the second master node in the PXE network.

      ac:1f:6b:02:84:72

      SET_MACHINE_1_BMC_ADDRESS

      The IP address of the BMC endpoint for the second master node in the cluster. Must be an address from the OOB network that is accessible through the management network gateway.

      192.168.100.12

      SET_MACHINE_2_IPMI_USERNAME

      The IPMI user name to access the BMC. 0

      user

      SET_MACHINE_2_IPMI_PASSWORD

      The IPMI password to access the BMC. 0

      password

      SET_MACHINE_2_MAC

      The MAC address of the third master node in the PXE network.

      ac:1f:6b:02:84:73

      SET_MACHINE_2_BMC_ADDRESS

      The IP address of the BMC endpoint for the third master node in the cluster. Must be an address from the OOB network that is accessible through the management network gateway.

      192.168.100.13

      0(1,2,3,4,5,6)

      The parameter requires a user name and password in plain text.

    4. Configure cluster network:

      Important

      Mirantis recommends separating the PXE and LCM networks.

      • To ensure successful bootstrap, enable asymmetric routing on the interfaces of the management cluster nodes. This is required because the seed node relies on one network by default, which can potentially cause traffic asymmetry.

        In the kernelParameters section of bm/baremetalhostprofiles.yaml.template, set rp_filter to 2. This enables loose mode as defined in RFC3704.

        Example configuration of asymmetric routing
        ...
        kernelParameters:
          ...
          sysctl:
            # Enables the "Loose mode" for the "k8s-lcm" interface (management network)
            net.ipv4.conf.k8s-lcm.rp_filter: "2"
            # Enables the "Loose mode" for the "bond0" interface (PXE network)
            net.ipv4.conf.bond0.rp_filter: "2"
            ...
        

        Note

        More complicated solutions that are not described in this manual include getting rid of traffic asymmetry, for example:

        • Configure source routing on management cluster nodes.

        • Plug the seed node into the same networks as the management cluster nodes, which requires custom configuration of the seed node.

      • Update the network objects definition in templates/bm/ipam-objects.yaml.template according to the environment configuration. By default, this template implies the use of separate PXE and life-cycle management (LCM) networks.

      • Manually set all parameters that start with SET_.

      Example of the default L2 template snippet for a management cluster:

      bonds:
        bond0:
          interfaces:
            - {{ nic 0 }}
            - {{ nic 1 }}
          parameters:
            mode: active-backup
            primary: {{ nic 0 }}
          dhcp4: false
          dhcp6: false
          addresses:
            - {{ ip "bond0:mgmt-pxe" }}
      vlans:
        k8s-lcm:
          id: SET_VLAN_ID
          link: bond0
          addresses:
            - {{ ip "k8s-lcm:kaas-mgmt" }}
          nameservers:
            addresses: {{ nameservers_from_subnet "kaas-mgmt" }}
          routes:
            - to: 0.0.0.0/0
              via: {{ gateway_from_subnet "kaas-mgmt" }}
      

      In this example, the following configuration applies:

      • A bond of two NIC interfaces

      • A static address in the PXE network set on the bond

      • An isolated L2 segment for the LCM network is configured using the k8s-lcm VLAN with the static address in the LCM network

      • The default gateway address is in the LCM network

      For general concepts of configuring separate PXE and LCM networks for a management cluster, see Separate PXE and management networks. For the latest object templates and variable names to use since Container Cloud 2.24.0, use the following tables.

      Network parameters mapping overview

      Deployment file name

      Parameters list to update manually

      ipam-objects.yaml.template

      • SET_LB_HOST

      • SET_MGMT_ADDR_RANGE

      • SET_MGMT_CIDR

      • SET_MGMT_DNS

      • SET_MGMT_NW_GW

      • SET_MGMT_SVC_POOL

      • SET_PXE_ADDR_POOL

      • SET_PXE_ADDR_RANGE

      • SET_PXE_CIDR

      • SET_PXE_SVC_POOL

      • SET_VLAN_ID

      bootstrap.env

      • KAAS_BM_PXE_IP

      • KAAS_BM_PXE_MASK

      • KAAS_BM_PXE_BRIDGE

      The below table contains examples of mandatory parameter values to set in templates/bm/ipam-objects.yaml.template for the network scheme that has the following networks:

      • 172.16.59.0/24 - PXE network

      • 172.16.61.0/25 - LCM network

      Mandatory network parameters of the IPAM objects template

      Parameter

      Description

      Example value

      SET_PXE_CIDR

      The IP address of the PXE network in the CIDR notation. The minimum recommended network size is 256 addresses (/24 prefix length).

      172.16.59.0/24

      SET_PXE_SVC_POOL

      The IP address range to use for endpoints of load balancers in the PXE network for the Container Cloud services: Ironic-API, DHCP server, HTTP server, and caching server. The minimum required range size is 5 addresses.

      172.16.59.6-172.16.59.15

      SET_PXE_ADDR_POOL

      The IP address range in the PXE network to use for dynamic address allocation for hosts during inspection and provisioning.

      The minimum recommended range size is 30 addresses for management cluster nodes if it is located in a separate PXE network segment. Otherwise, it depends on the number of managed cluster nodes to deploy in the same PXE network segment as the management cluster nodes.

      172.16.59.51-172.16.59.200

      SET_PXE_ADDR_RANGE

      The IP address range in the PXE network to use for static address allocation on each management cluster node. The minimum recommended range size is 6 addresses.

      172.16.59.41-172.16.59.50

      SET_MGMT_CIDR

      The IP address of the LCM network for the management cluster in the CIDR notation. If managed clusters will have their separate LCM networks, those networks must be routable to the LCM network. The minimum recommended network size is 128 addresses (/25 prefix length).

      172.16.61.0/25

      SET_MGMT_NW_GW

      The default gateway address in the LCM network. This gateway must provide access to the OOB network of the Container Cloud cluster and to the Internet to download the Mirantis artifacts.

      172.16.61.1

      SET_LB_HOST

      The IP address of the externally accessible MKE API endpoint of the cluster in the CIDR notation. This address must be within the management SET_MGMT_CIDR network but must NOT overlap with any other addresses or address ranges within this network. External load balancers are not supported.

      172.16.61.5/32

      SET_MGMT_DNS

      An external (non-Kubernetes) DNS server accessible from the LCM network.

      8.8.8.8

      SET_MGMT_ADDR_RANGE

      The IP address range that includes addresses to be allocated to bare metal hosts in the LCM network for the management cluster.

      When this network is shared with managed clusters, the size of this range limits the number of hosts that can be deployed in all clusters sharing this network.

      When this network is solely used by a management cluster, the range must include at least 6 addresses for bare metal hosts of the management cluster.

      172.16.61.30-172.16.61.40

      SET_MGMT_SVC_POOL

      The IP address range to use for the externally accessible endpoints of load balancers in the LCM network for the Container Cloud services, such as Keycloak, web UI, and so on. The minimum required range size is 19 addresses.

      172.16.61.10-172.16.61.29

      SET_VLAN_ID

      The VLAN ID used for isolation of LCM network. The bootstrap.sh process and the seed node must have routable access to the network in this VLAN.

      3975

      When using separate PXE and LCM networks, the management cluster services are exposed in different networks using two separate MetalLB address pools:

      • Services exposed through the PXE network are as follows:

        • Ironic API as a bare metal provisioning server

        • HTTP server that provides images for network boot and server provisioning

        • Caching server for accessing the Container Cloud artifacts deployed on hosts

      • Services exposed through the LCM network are all other Container Cloud services, such as Keycloak, web UI, and so on.

      The default MetalLB configuration described in the MetalLBConfigTemplate object template of templates/bm/ipam-objects.yaml.template uses two separate MetalLB address pools. Also, it uses the interfaces selector in its l2Advertisements template.

      Caution

      When you change the L2Template object template in templates/bm/ipam-objects.yaml.template, ensure that interfaces listed in the interfaces field of the MetalLBConfigTemplate.spec.templates.l2Advertisements section match those used in your L2Template. For details about the interfaces selector, see API Reference: MetalLBConfigTemplate spec.

      See Configure MetalLB for details on MetalLB configuration.

    5. If you require all Internet traffic to go through a proxy server, in bootstrap.env, add the following environment variables to bootstrap the cluster using proxy:

      • HTTP_PROXY

      • HTTPS_PROXY

      • NO_PROXY

      • PROXY_CA_CERTIFICATE_PATH

      Example snippet:

      export HTTP_PROXY=http://proxy.example.com:3128
      export HTTPS_PROXY=http://user:pass@proxy.example.com:3128
      export NO_PROXY=172.18.10.0,registry.internal.lan
      export PROXY_CA_CERTIFICATE_PATH="/home/ubuntu/.mitmproxy/mitmproxy-ca-cert.cer"
      

      The following formats of variables are accepted:

      Proxy configuration data

      Variable

      Format

      HTTP_PROXY
      HTTPS_PROXY
      • http://proxy.example.com:port - for anonymous access.

      • http://user:password@proxy.example.com:port - for restricted access.

      NO_PROXY

      Comma-separated list of IP addresses or domain names.

      PROXY_CA_CERTIFICATE_PATH

      Optional. Absolute path to the proxy CA certificate for man-in-the-middle (MITM) proxies. Must be placed on the bootstrap node to be trusted. For details, see Install a CA certificate for a MITM proxy on a bootstrap node.

      Warning

      If you require Internet access to go through a MITM proxy, ensure that the proxy has streaming enabled as described in Enable streaming for MITM.

      For implementation details, see Proxy and cache support.

      For the requirements on Mirantis resources and IP ranges that must be accessible from the bootstrap, management, and regional clusters, see Requirements for a baremetal-based cluster.

      For the default network addresses used by Swarm, see Default network addresses.

    6. Inspect the templates/bm/machines.yaml.template and adjust spec and labels of each entry according to your deployment. Adjust spec.providerSpec.value.hostSelector values to match BareMetalHost corresponding to each machine. For details, see API Reference: Bare metal Machine spec.

    7. In templates/bm/cluster.yaml.template, update the cluster-related settings to fit your deployment.

    8. Configure NTP server.

      You can optionally disable NTP that is enabled by default. This option disables the management of chrony configuration by Container Cloud to use your own system for chrony management. Otherwise, configure the regional NTP server parameters as described below.

      NTP configuration

      Configure the regional NTP server parameters to be applied to all machines of regional and managed clusters in the specified region.

      In templates/bm/cluster.yaml.template, add the ntp:servers section with the list of required server names:

      spec:
        ...
        providerSpec:
          value:
            kaas:
            ...
            ntpEnabled: true
              regional:
                - helmReleases:
                  - name: <providerName>-provider
                    values:
                      config:
                        lcm:
                          ...
                          ntp:
                            servers:
                            - 0.pool.ntp.org
                            ...
                  provider: <providerName>
                  ...
      

      To disable NTP:

      spec:
        ...
        providerSpec:
          value:
            ...
            ntpEnabled: false
            ...
      
    9. Verify that the kaas-bootstrap directory contains the following files:

      # tree  ~/kaas-bootstrap
        ~/kaas-bootstrap/
        ....
        ├── bootstrap.sh
        ├── kaas
        ├── mirantis.lic
        ├── releases
        ...
        ├── templates
        ....
                     ├── bm
                                  ├── baremetalhostprofiles.yaml.template
                                  ├── baremetalhosts.yaml.template
                                  ├── cluster.yaml.template
                                  ├── ipam-objects.yaml.template
                                  └── machines.yaml.template
                                  └── metallbconfig.yaml.template
        ....
        ├── templates.backup
            ....
      
    10. Export the following mandatory parameters using the commands and table below:

      export KAAS_BM_ENABLED="true"
      #
      export KAAS_BM_PXE_IP="172.16.59.5"
      export KAAS_BM_PXE_MASK="24"
      export KAAS_BM_PXE_BRIDGE="br0"
      
      Bare metal prerequisites data

      Parameter

      Description

      Example value

      KAAS_BM_PXE_IP

      The provisioning IP address in the PXE network. This address will be assigned on the seed node to the interface defined by the KAAS_BM_PXE_BRIDGE parameter described below. The PXE service of the bootstrap cluster uses this address to network boot bare metal hosts.

      172.16.59.5

      KAAS_BM_PXE_MASK

      The PXE network address prefix length to be used with the KAAS_BM_PXE_IP address when assigning it to the seed node interface.

      24

      KAAS_BM_PXE_BRIDGE

      The PXE network bridge name that must match the name of the bridge created on the seed node during the Prepare the seed node stage.

      br0

    11. Unset the deprecated KAAS_BM_FULL_PREFLIGHT parameter:

      unset KAAS_BM_FULL_PREFLIGHT
      
    12. Optional. Enable WireGuard for traffic encryption on the Kubernetes workloads network.

      WireGuard configuration
      1. Ensure that the Calico MTU size is at least 60 bytes smaller than the interface MTU size of the workload network. IPv4 WireGuard uses a 60-byte header. For details, see Set the MTU size for Calico.

      2. In templates/bm/cluster.yaml.template, enable WireGuard by adding the secureOverlay parameter:

        spec:
          ...
          providerSpec:
            value:
              ...
              secureOverlay: true
        

        Caution

        Changing this parameter on a running cluster causes a downtime that can vary depending on the cluster size.

      For more details about WireGuard, see Calico documentation: Encrypt in-cluster pod traffic.

    13. Technology Preview. Enable custom host names for cluster machines. When enabled, any machine host name in a particular region matches the related Machine object name. For example, instead of the default kaas-node-<UID>, a machine host name will be master-0. The custom naming format is more convenient and easier to operate with.

      To enable the feature on the management and its future managed clusters:

      1. In templates/bm/cluster.yaml.template, find the spec.providerSpec.value.kaas.regional section of the required region.

      2. In this section, find the required provider name under helmReleases.

      3. Under values.config, add customHostnamesEnabled: true.

        For example, for the bare metal provider in region-one:

        regional:
         - helmReleases:
           - name: baremetal-provider
             values:
               config:
                 allInOneAllowed: false
                 customHostnamesEnabled: true
                 internalLoadBalancers: false
           provider: baremetal-provider
        

      Add the following environment variable:

      export CUSTOM_HOSTNAMES=true
      
    14. Optional. Configure external identity provider for IAM.

    15. Optional. Enable infinite timeout for all bootstrap stages by exporting the following environment variable or adding it to bootstrap.env:

      export KAAS_BOOTSTRAP_INFINITE_TIMEOUT=true
      

      Infinite timeout prevents the bootstrap failure due to timeout. This option is useful in the following cases:

      • The network speed is slow for artifacts downloading

      • An infrastructure configuration does not allow booting fast

      • A bare-metal node inspecting presupposes more than two HDDSATA disks to attach to a machine

    16. Optional. Available since Container Cloud 2.23.0. Customize the cluster and region name by exporting the following environment variables or adding them to bootstrap.env:

      export REGION=<customRegionName>
      export CLUSTER_NAME=<customClusterName>
      

      By default, the system uses region-one for the region name and kaas-mgmt for the management cluster name.

  6. Run the bootstrap script:

    ./bootstrap.sh all
    
    • In case of deployment issues, refer to Troubleshooting and inspect logs.

    • If the script fails for an unknown reason:

      1. Run the cleanup script:

        ./bootstrap.sh cleanup
        
      2. Rerun the bootstrap script.

    Warning

    During the bootstrap process, do not manually restart or power off any of the bare metal hosts.

  7. When the bootstrap is complete, collect and save the following management cluster details in a secure location:

    • The kubeconfig file located in the same directory as the bootstrap script. This file contains the admin credentials for the management cluster.

    • The private ssh_key for access to the management cluster nodes that is located in the same directory as the bootstrap script.

      Note

      If the initial version of your Container Cloud management cluster was earlier than 2.6.0, ssh_key is named openstack_tmp and is located at ~/.ssh/.

    • The URL for the Container Cloud web UI.

      To create users with permissions required for accessing the Container Cloud web UI, see Create initial users after a management cluster bootstrap.

    • The StackLight endpoints. For details, see Access StackLight web UIs.

    • The Keycloak URL that the system outputs when the bootstrap completes. The admin password for Keycloak is located in kaas-bootstrap/passwords.yml along with other IAM passwords.

    Note

    The Container Cloud web UI and StackLight endpoints are available through Transport Layer Security (TLS) and communicate with Keycloak to authenticate users. Keycloak is exposed using HTTPS and self-signed TLS certificates that are not trusted by web browsers.

    To use your own TLS certificates for Keycloak, refer to Configure TLS certificates for cluster applications.

    Note

    When the bootstrap is complete, the bootstrap cluster resources are freed up.

  8. Verify that network addresses used on your clusters do not overlap with the following default MKE network addresses for Swarm and MCR:

    • 10.0.0.0/16 is used for Swarm networks. IP addresses from this network are virtual.

    • 10.99.0.0/16 is used for MCR networks. IP addresses from this network are allocated on hosts.

    Verification of Swarm and MCR network addresses

    To verify Swarm and MCR network addresses, run on any master node:

    docker info
    

    Example of system response:

    Server:
     ...
     Swarm:
      ...
      Default Address Pool: 10.0.0.0/16
      SubnetSize: 24
      ...
     Default Address Pools:
       Base: 10.99.0.0/16, Size: 20
     ...
    

    Not all of Swarm and MCR addresses are usually in use. One Swarm Ingress network is created by default and occupies the 10.0.0.0/24 address block. Also, three MCR networks are created by default and occupy three address blocks: 10.99.0.0/20, 10.99.16.0/20, 10.99.32.0/20.

    To verify the actual networks state and addresses in use, run:

    docker network ls
    docker network inspect <networkName>
    
  9. Optional. If you plan to use multiple L2 segments for provisioning of managed cluster nodes, consider the requirements specified in Configure multiple DHCP ranges using Subnet resources.

  10. Optional. Deploy an additional regional cluster of a different provider type as described in Deploy an additional regional cluster (optional).

Prepare metadata and deploy the management cluster before 2.24.0

This section describes how to prepare cluster metadata and deploy the management cluster before Container Cloud 2.24.0. The section also contains instructions on additional advanced configuration that you may require to customize your cluster.

Prepare metadata and deploy the management cluster

This section describes how to prepare cluster metadata and deploy the management cluster before Container Cloud 2.24.0. For the procedure that applies since 2.24.0, refer to Prepare metadata and deploy the management cluster since 2.24.0.

Using the example procedure below, replace the addresses and credentials in the configuration YAML files with the data from your environment. Keep everything else as is, including the file names and YAML structure.

The overall network mapping scheme with all L2/L3 parameters, for example, for a single 10.0.0.0/24 network, is described in the following table. The configuration of each parameter included in this table is described in the procedure below.

Network mapping overview

Deployment file name

Parameters and values

cluster.yaml

  • SET_LB_HOST=10.0.0.90

  • SET_METALLB_ADDR_POOL=10.0.0.61-10.0.0.80

ipam-objects.yaml

  • SET_IPAM_CIDR=10.0.0.0/24

  • SET_PXE_NW_GW=10.0.0.1

  • SET_PXE_NW_DNS=8.8.8.8

  • SET_IPAM_POOL_RANGE=10.0.0.100-10.0.0.252

  • SET_LB_HOST=10.0.0.90

  • SET_METALLB_ADDR_POOL=10.0.0.61-10.0.0.80

bootstrap.env

  • KAAS_BM_PXE_IP=10.0.0.20

  • KAAS_BM_PXE_MASK=24

  • KAAS_BM_PXE_BRIDGE=br0

  • KAAS_BM_BM_DHCP_RANGE=10.0.0.30,10.0.0.49,255.255.255.0

  • BOOTSTRAP_METALLB_ADDRESS_POOL=10.0.0.61-10.0.0.80

To prepare metadata and deploy the management cluster:

  1. Log in to the seed node that you configured as described in Prepare the seed node.

  2. Change to your preferred work directory, for example, your home directory:

    cd $HOME
    
  3. Prepare the bootstrap script:

    1. Download and run the Container Cloud bootstrap script:

      sudo apt-get update
      sudo apt-get install wget
      wget https://binary.mirantis.com/releases/get_container_cloud.sh
      chmod 0755 get_container_cloud.sh
      ./get_container_cloud.sh
      
    2. Change the directory to the kaas-bootstrap folder created by the script.

  4. Obtain your license file that will be required during the bootstrap:

    1. Select from the following options:

      • Open the email from support@mirantis.com with the subject Mirantis Container Cloud License File or Mirantis OpenStack License File

      • In the Mirantis CloudCare Portal, open the Account or Cloud page

    2. Download the License File and save it as mirantis.lic under the kaas-bootstrap directory on the bootstrap node.

    3. Verify that mirantis.lic contains the previously downloaded Container Cloud license by decoding the license JWT token, for example, using jwt.io.

      Example of a valid decoded Container Cloud license data with the mandatory license field:

      {
          "exp": 1652304773,
          "iat": 1636669973,
          "sub": "demo",
          "license": {
              "dev": false,
              "limits": {
                  "clusters": 10,
                  "workers_per_cluster": 10
              },
              "openstack": null
          }
      }
      

      Warning

      The MKE license does not apply to mirantis.lic. For details about MKE license, see MKE documentation.

  5. Prepare the deployment templates:

    Note

    The serviceusers.yaml.template and bootstrapregion.yaml.template files relate to the Bootstrap v2 procedure only. Therefore, skip these templates in the Bootstrap v1 deployments.

    For details on the Bootstrap v2 procedure, see Deploy Container Cloud using Boostrap v2.

    1. Create a copy of the current templates directory for future reference.

      mkdir templates.backup
      cp -r templates/*  templates.backup/
      
    2. Update the cluster definition template in templates/bm/cluster.yaml.template according to the environment configuration. Use the table below. Manually set all parameters that start with SET_. For example, SET_METALLB_ADDR_POOL.

      Cluster template mandatory parameters

      Parameter

      Description

      Example value

      SET_LB_HOST

      The IP address of the externally accessible API endpoint of the cluster. This address must NOT be within the SET_METALLB_ADDR_POOL range but must be within the PXE/Management network. External load balancers are not supported.

      10.0.0.90

      SET_METALLB_ADDR_POOL

      The IP range to be used as external load balancers for the Kubernetes services with the LoadBalancer type. This range must be within the PXE/Management network. The minimum required range is 19 IP addresses.

      10.0.0.61-10.0.0.80

    3. Configure NTP server.

      Before Container Cloud 2.23.0, optional if servers from the Ubuntu NTP pool (*.ubuntu.pool.ntp.org) are accessible from the node where your cluster is being provisioned. Otherwise, configure the regional NTP server parameters as described below.

      Since Container Cloud 2.23.0, optionally disable NTP that is enabled by default. This option disables the management of chrony configuration by Container Cloud to use your own system for chrony management. Otherwise, configure the regional NTP server parameters as described below.

      NTP configuration

      Configure the regional NTP server parameters to be applied to all machines of regional and managed clusters in the specified region.

      In templates/bm/cluster.yaml.template, add the ntp:servers section with the list of required server names:

      spec:
        ...
        providerSpec:
          value:
            kaas:
            ...
            ntpEnabled: true
              regional:
                - helmReleases:
                  - name: <providerName>-provider
                    values:
                      config:
                        lcm:
                          ...
                          ntp:
                            servers:
                            - 0.pool.ntp.org
                            ...
                  provider: <providerName>
                  ...
      

      To disable NTP:

      spec:
        ...
        providerSpec:
          value:
            ...
            ntpEnabled: false
            ...
      
    4. Inspect the default bare metal host profile definition in templates/bm/baremetalhostprofiles.yaml.template. If your hardware configuration differs from the reference, adjust the default profile to match. For details, see Customize the default bare metal host profile.

      Warning

      All data will be wiped during cluster deployment on devices defined directly or indirectly in the fileSystems list of BareMetalHostProfile. For example:

      • A raw device partition with a file system on it

      • A device partition in a volume group with a logical volume that has a file system on it

      • An mdadm RAID device with a file system on it

      • An LVM RAID device with a file system on it

      The wipe field is always considered true for these devices. The false value is ignored.

      Therefore, to prevent data loss, move the necessary data from these file systems to another server beforehand, if required.

    5. In templates/bm/baremetalhosts.yaml.template, update the bare metal host definitions according to your environment configuration. Use the table below for reference.

      • Manually set all parameters that start with SET_.

      • Set the |region-value| for the kaas.mirantis.com/region label across all objects listed in templates/bm/baremetalhosts.yaml.template.

      Bare metal hosts template mandatory parameters

      Parameter

      Description

      Example value

      SET_MACHINE_0_IPMI_USERNAME

      The IPMI user name to access the BMC. 0

      user

      SET_MACHINE_0_IPMI_PASSWORD

      The IPMI password to access the BMC. 0

      password

      SET_MACHINE_0_MAC

      The MAC address of the first master node in the PXE network.

      ac:1f:6b:02:84:71

      SET_MACHINE_0_BMC_ADDRESS

      The IP address of the BMC endpoint for the first master node in the cluster. Must be an address from the OOB network that is accessible through the management network gateway.

      192.168.100.11

      SET_MACHINE_1_IPMI_USERNAME

      The IPMI user name to access the BMC. 0

      user

      SET_MACHINE_1_IPMI_PASSWORD

      The IPMI password to access the BMC. 0

      password

      SET_MACHINE_1_MAC

      The MAC address of the second master node in the PXE network.

      ac:1f:6b:02:84:72

      SET_MACHINE_1_BMC_ADDRESS

      The IP address of the BMC endpoint for the second master node in the cluster. Must be an address from the OOB network that is accessible through the management network gateway.

      192.168.100.12

      SET_MACHINE_2_IPMI_USERNAME

      The IPMI user name to access the BMC. 0

      user

      SET_MACHINE_2_IPMI_PASSWORD

      The IPMI password to access the BMC. 0

      password

      SET_MACHINE_2_MAC

      The MAC address of the third master node in the PXE network.

      ac:1f:6b:02:84:73

      SET_MACHINE_2_BMC_ADDRESS

      The IP address of the BMC endpoint for the third master node in the cluster. Must be an address from the OOB network that is accessible through the management network gateway.

      192.168.100.13

      0(1,2,3,4,5,6)

      The parameter requires a user name and password in plain text.

    6. Update the Subnet objects definition template in templates/bm/ipam-objects.yaml.template according to the environment configuration. Use the table below. Manually set all parameters that start with SET_. For example, SET_IPAM_POOL_RANGE.

      IP address pools template mandatory parameters

      Parameter

      Description

      Example value

      SET_IPAM_CIDR

      The address of PXE network in CIDR notation. Must be minimum in the /24 network.

      10.0.0.0/24

      SET_PXE_NW_GW

      The default gateway in the PXE network. Since this is the only network that cluster will use by default, this gateway must provide access to:

      • The Internet to download the Mirantis artifacts

      • The OOB network of the Container Cloud cluster

      10.0.0.1

      SET_PXE_NW_DNS

      An external (non-Kubernetes) DNS server accessible from the PXE network.

      8.8.8.8

      SET_IPAM_POOL_RANGE

      This IP address range includes addresses that will be allocated in the PXE/Management network to bare metal hosts of the cluster.

      10.0.0.100-10.0.0.252

      SET_LB_HOST 1

      The IP address of the externally accessible API endpoint of the cluster. This address must NOT be within the SET_METALLB_ADDR_POOL range but must be within the PXE/Management network. External load balancers are not supported.

      10.0.0.90

      SET_METALLB_ADDR_POOL 1

      The IP address range to be used as external load balancers for the Kubernetes services with the LoadBalancer type. This range must be within the PXE/Management network. The minimum required range is 19 IP addresses.

      10.0.0.61-10.0.0.80

      1(1,2)

      Use the same value that you used for this parameter in the cluster.yaml.template file (see above).

    7. Optional. To configure the separated PXE and management networks instead of one PXE/management network, proceed to Separate PXE and management networks.

    8. Optional. To connect the cluster hosts to the PXE/Management network using bond interfaces, proceed to Configure NIC bonding.

    9. If you require all Internet access to go through a proxy server, in bootstrap.env, add the following environment variables to bootstrap the cluster using proxy:

      • HTTP_PROXY

      • HTTPS_PROXY

      • NO_PROXY

      • PROXY_CA_CERTIFICATE_PATH

      Example snippet:

      export HTTP_PROXY=http://proxy.example.com:3128
      export HTTPS_PROXY=http://user:pass@proxy.example.com:3128
      export NO_PROXY=172.18.10.0,registry.internal.lan
      export PROXY_CA_CERTIFICATE_PATH="/home/ubuntu/.mitmproxy/mitmproxy-ca-cert.cer"
      

      The following formats of variables are accepted:

      Proxy configuration data

      Variable

      Format

      HTTP_PROXY
      HTTPS_PROXY
      • http://proxy.example.com:port - for anonymous access.

      • http://user:password@proxy.example.com:port - for restricted access.

      NO_PROXY

      Comma-separated list of IP addresses or domain names.

      PROXY_CA_CERTIFICATE_PATH

      Optional. Absolute path to the proxy CA certificate for man-in-the-middle (MITM) proxies. Must be placed on the bootstrap node to be trusted. For details, see Install a CA certificate for a MITM proxy on a bootstrap node.

      Warning

      If you require Internet access to go through a MITM proxy, ensure that the proxy has streaming enabled as described in Enable streaming for MITM.

      For implementation details, see Proxy and cache support.

      For the list of Mirantis resources and IP addresses to be accessible from the Container Cloud clusters, see Requirements for a baremetal-based cluster.

    10. Available since Container Cloud 2.24.0 and 2.24.2 for MOSK 23.2. Optional. Technology Preview. Enable the Linux Audit daemon auditd to monitor activity of cluster processes and prevent potential malicious activity.

      Configuration for auditd

      In templates/bm/cluster.yaml.template, add the auditd parameters:

      spec:
        providerSpec:
          value:
            audit:
              auditd:
                enabled: <bool>
                enabledAtBoot: <bool>
                backlogLimit: <int>
                maxLogFile: <int>
                maxLogFileAction: <string>
                maxLogFileKeep: <int>
                mayHaltSystem: <bool>
                presetRules: <string>
                customRules: <string>
                customRulesX32: <text>
                customRulesX64: <text>
      

      Configuration parameters for auditd:

      enabled

      Boolean, default - false. Enables the auditd role to install the auditd packages and configure rules. CIS rules: 4.1.1.1, 4.1.1.2.

      enabledAtBoot

      Boolean, default - false. Configures grub to audit processes that can be audited even if they start up prior to auditd startup. CIS rule: 4.1.1.3.

      backlogLimit

      Integer, default - none. Configures the backlog to hold records. If during boot audit=1 is configured, the backlog holds 64 records. If more than 64 records are created during boot, auditd records will be lost with a potential malicious activity being undetected. CIS rule: 4.1.1.4.

      maxLogFile

      Integer, default - none. Configures the maximum size of the audit log file. Once the log reaches the maximum size, it is rotated and a new log file is created. CIS rule: 4.1.2.1.

      maxLogFileAction

      String, default - none. Defines handling of the audit log file reaching the maximum file size. Allowed values:

      • keep_logs - rotate logs but never delete them

      • rotate - add a cron job to compress rotated log files and keep maximum 5 compressed files.

      • compress - compress log files and keep them under the /var/log/auditd/ directory. Requires auditd_max_log_file_keep to be enabled.

      CIS rule: 4.1.2.2.

      maxLogFileKeep

      Integer, default - 5. Defines the number of compressed log files to keep under the /var/log/auditd/ directory. Requires auditd_max_log_file_action=compress. CIS rules - none.

      mayHaltSystem

      Boolean, default - false. Halts the system when the audit logs are full. Applies the following configuration:

      • space_left_action = email

      • action_mail_acct = root

      • admin_space_left_action = halt

      CIS rule: 4.1.2.3.

      customRules

      String, default - none. Base64-encoded content of the 60-custom.rules file for any architecture. CIS rules - none.

      customRulesX32

      String, default - none. Base64-encoded content of the 60-custom.rules file for the i386 architecture. CIS rules - none.

      customRulesX64

      String, default - none. Base64-encoded content of the 60-custom.rules file for the x86_64 architecture. CIS rules - none.

      presetRules

      String, default - none. Comma-separated list of the following built-in preset rules:

      • access

      • actions

      • delete

      • docker

      • identity

      • immutable

      • logins

      • mac-policy

      • modules

      • mounts

      • perm-mod

      • privileged

      • scope

      • session

      • system-locale

      • time-change

      You can use two keywords for these rules:

      • none - disables all built-in rules.

      • all - enables all built-in rules. With this key, you can add the ! prefix to a rule name to exclude some rules. You can use the ! prefix for rules only if you add the all keyword as the first rule. Place a rule with the ! prefix only after the all keyword.

      Example configurations:

      • presetRules: none - disable all preset rules

      • presetRules: docker - enable only the docker rules

      • presetRules: access,actions,logins - enable only the access, actions, and logins rules

      • presetRules: all - enable all preset rules

      • presetRules: all,!immutable,!sessions - enable all preset rules except immutable and sessions


      CIS controls
      4.1.3 (time-change)
      4.1.4 (identity)
      4.1.5 (system-locale)
      4.1.6 (mac-policy)
      4.1.7 (logins)
      4.1.8 (session)
      4.1.9 (perm-mod)
      4.1.10 (access)
      4.1.11 (privileged)
      4.1.12 (mounts)
      4.1.13 (delete)
      4.1.14 (scope)
      4.1.15 (actions)
      4.1.16 (modules)
      4.1.17 (immutable)
      Docker CIS controls
      1.1.4
      1.1.8
      1.1.10
      1.1.12
      1.1.13
      1.1.15
      1.1.16
      1.1.17
      1.1.18
      1.2.3
      1.2.4
      1.2.5
      1.2.6
      1.2.7
      1.2.10
      1.2.11
    11. Available as Technology Preview since 2.24.0 and 2.24.2 for MOSK 23.2. Optional. Enable WireGuard for traffic encryption on the Kubernetes workloads network.

      WireGuard configuration
      1. Ensure that the Calico MTU size is at least 60 bytes smaller than the interface MTU size of the workload network. IPv4 WireGuard uses a 60-byte header. For details, see Set the MTU size for Calico.

      2. In templates/bm/cluster.yaml.template, enable WireGuard by adding the secureOverlay parameter:

        spec:
          ...
          providerSpec:
            value:
              ...
              secureOverlay: true
        

        Caution

        Changing this parameter on a running cluster causes a downtime that can vary depending on the cluster size.

      For more details about WireGuard, see Calico documentation: Encrypt in-cluster pod traffic.

    12. Available since Container Cloud 2.24.0. Optional. Technology Preview. Enable custom host names for cluster machines. When enabled, any machine host name in a particular region matches the related Machine object name. For example, instead of the default kaas-node-<UID>, a machine host name will be master-0. The custom naming format is more convenient and easier to operate with.

      To enable the feature on the management and its future managed clusters:

      1. In templates/bm/cluster.yaml.template, find the spec.providerSpec.value.kaas.regional section of the required region.

      2. In this section, find the required provider name under helmReleases.

      3. Under values.config, add customHostnamesEnabled: true.

        For example, for the bare metal provider in region-one:

        regional:
         - helmReleases:
           - name: baremetal-provider
             values:
               config:
                 allInOneAllowed: false
                 customHostnamesEnabled: true
                 internalLoadBalancers: false
           provider: baremetal-provider
        

      Add the following environment variable:

      export CUSTOM_HOSTNAMES=true
      
    13. Verify that the kaas-bootstrap directory contains the following files:

      # tree  ~/kaas-bootstrap
        ~/kaas-bootstrap/
        ....
        ├── bootstrap.sh
        ├── kaas
        ├── mirantis.lic
        ├── releases
        ...
        ├── templates
        ....
                     ├── bm
                                  ├── baremetalhostprofiles.yaml.template
                                  ├── baremetalhosts.yaml.template
                                  ├── cluster.yaml.template
                                  ├── ipam-objects.yaml.template
                                  └── machines.yaml.template
        ....
        ├── templates.backup
            ....
      
    14. Export all required parameters using the table below.

      export KAAS_BM_ENABLED="true"
      #
      export KAAS_BM_PXE_IP="10.0.0.20"
      export KAAS_BM_PXE_MASK="24"
      export KAAS_BM_PXE_BRIDGE="br0"
      #
      export KAAS_BM_BM_DHCP_RANGE="10.0.0.30,10.0.0.49,255.255.255.0"
      export BOOTSTRAP_METALLB_ADDRESS_POOL="10.0.0.61-10.0.0.80"
      #
      unset KAAS_BM_FULL_PREFLIGHT
      
      Bare metal prerequisites data

      Parameter

      Description

      Example value

      KAAS_BM_PXE_IP

      The provisioning IP address. This address will be assigned to the interface of the seed node defined by the KAAS_BM_PXE_BRIDGE parameter (see below). The PXE service of the bootstrap cluster will use this address to network boot the bare metal hosts for the cluster.

      10.0.0.20

      KAAS_BM_PXE_MASK

      The CIDR prefix for the PXE network. It will be used with KAAS_BM_PXE_IP address when assigning it to network interface.

      24

      KAAS_BM_PXE_BRIDGE

      The PXE network bridge name. The name must match the name of the bridge created on the seed node during the Prepare the seed node stage.

      br0

      KAAS_BM_BM_DHCP_RANGE

      The start_ip and end_ip addresses must be within the PXE network. This range will be used by dnsmasq to provide IP addresses for nodes during provisioning.

      10.0.0.30,10.0.0.49,255.255.255.0

      BOOTSTRAP_METALLB_ADDRESS_POOL

      The pool of IP addresses that will be used by services in the bootstrap cluster. Can be the same as the SET_METALLB_ADDR_POOL range for the cluster, or a different range.

      10.0.0.61-10.0.0.80

    15. Run the verification preflight script to validate the deployment templates configuration:

      ./bootstrap.sh preflight
      

      The command outputs a human-readable report with the verification details. The report includes the list of verified bare metal nodes and their Chassis Power status. This status is based on the deployment templates configuration used during the verification.

      Caution

      If the report contains information about missing dependencies or incorrect configuration, fix the issues before proceeding to the next step.

  6. Optional. Configure external identity provider for IAM.

  7. Optional. Enable infinite timeout for all bootstrap stages by exporting the following environment variable or adding it to bootstrap.env:

    export KAAS_BOOTSTRAP_INFINITE_TIMEOUT=true
    

    Infinite timeout prevents the bootstrap failure due to timeout. This option is useful in the following cases:

    • The network speed is slow for artifacts downloading

    • An infrastructure configuration does not allow booting fast

    • A bare-metal node inspecting presupposes more than two HDDSATA disks to attach to a machine

  8. Optional. Available since Container Cloud 2.23.0. Customize the cluster and region name by exporting the following environment variables or adding them to bootstrap.env:

    export REGION=<customRegionName>
    export CLUSTER_NAME=<customClusterName>
    

    By default, the system uses region-one for the region name and kaas-mgmt for the management cluster name.

  9. Run the bootstrap script:

    ./bootstrap.sh all
    
    • In case of deployment issues, refer to Troubleshooting and inspect logs.

    • If the script fails for an unknown reason:

      1. Run the cleanup script:

        ./bootstrap.sh cleanup
        
      2. Rerun the bootstrap script.

    Warning

    During the bootstrap process, do not manually restart or power off any of the bare metal hosts.

  10. When the bootstrap is complete, collect and save the following management cluster details in a secure location:

    • The kubeconfig file located in the same directory as the bootstrap script. This file contains the admin credentials for the management cluster.

    • The private ssh_key for access to the management cluster nodes that is located in the same directory as the bootstrap script.

      Note

      If the initial version of your Container Cloud management cluster was earlier than 2.6.0, ssh_key is named openstack_tmp and is located at ~/.ssh/.

    • The URL for the Container Cloud web UI.

      To create users with permissions required for accessing the Container Cloud web UI, see Create initial users after a management cluster bootstrap.

    • The StackLight endpoints. For details, see Access StackLight web UIs.

    • The Keycloak URL that the system outputs when the bootstrap completes. The admin password for Keycloak is located in kaas-bootstrap/passwords.yml along with other IAM passwords.

    Note

    The Container Cloud web UI and StackLight endpoints are available through Transport Layer Security (TLS) and communicate with Keycloak to authenticate users. Keycloak is exposed using HTTPS and self-signed TLS certificates that are not trusted by web browsers.

    To use your own TLS certificates for Keycloak, refer to Configure TLS certificates for cluster applications.

    Note

    When the bootstrap is complete, the bootstrap cluster resources are freed up.

  11. Verify that network addresses used on your clusters do not overlap with the following default MKE network addresses for Swarm and MCR:

    • 10.0.0.0/16 is used for Swarm networks. IP addresses from this network are virtual.

    • 10.99.0.0/16 is used for MCR networks. IP addresses from this network are allocated on hosts.

    Verification of Swarm and MCR network addresses

    To verify Swarm and MCR network addresses, run on any master node:

    docker info
    

    Example of system response:

    Server:
     ...
     Swarm:
      ...
      Default Address Pool: 10.0.0.0/16
      SubnetSize: 24
      ...
     Default Address Pools:
       Base: 10.99.0.0/16, Size: 20
     ...
    

    Not all of Swarm and MCR addresses are usually in use. One Swarm Ingress network is created by default and occupies the 10.0.0.0/24 address block. Also, three MCR networks are created by default and occupy three address blocks: 10.99.0.0/20, 10.99.16.0/20, 10.99.32.0/20.

    To verify the actual networks state and addresses in use, run:

    docker network ls
    docker network inspect <networkName>
    
  12. Optional. If you plan to use multiple L2 segments for provisioning of managed cluster nodes, consider the requirements specified in Configure multiple DHCP ranges using Subnet resources.

  13. Optional. Deploy an additional regional cluster of a different provider type as described in Deploy an additional regional cluster (optional).

Configure NIC bonding

You can configure L2 templates for the management cluster to set up a bond network interface for the PXE/management network.

This configuration must be applied to the bootstrap templates, before you run the bootstrap script to deploy the management cluster.

Caution

  • This configuration requires each host in your management cluster to have at least two physical interfaces.

  • Connect at least two interfaces per host to an Ethernet switch that supports Link Aggregation Control Protocol (LACP) port groups and LACP fallback.

  • Configure an LACP group on the ports connected to the NICs of a host.

  • Configure the LACP fallback on the port group to ensure that the host can boot over the PXE network before the bond interface is set up on the host operating system.

  • Configure server BIOS for both NICs of a bond to be PXE-enabled.

  • If the server does not support booting from multiple NICs, configure the port of the LACP group that is connected to the PXE-enabled NIC of a server to be primary port. With this setting, the port becomes active in the fallback mode.

To configure a bond interface that aggregates two interfaces for the PXE/Management network:

  1. In kaas-bootstrap/templates/bm/ipam-objects.yaml.template:

    1. Configure only the following parameters for the declaration of {{nic 0}}, as shown in the example below:

      • dhcp4

      • dhcp6

      • match

      • set-name

      Remove other parameters.

    2. Add the declaration of the second NIC {{nic 1}} to be added to the bond interface:

      • Specify match:macaddress: {{mac 1}} to match the MAC of the desired NIC.

      • Specify set-name: {{nic 1}} to ensure the correct name of the NIC.

    3. Add the declaration of the bond interface bond0. It must have the interfaces parameter listing both Ethernet interfaces.

    4. Set the interfaces parameter of the management network bridge (k8s-lcm in the example below) to include bond0.

      Each node of every cluster must have only one IP address in the LCM network that is allocated from one of the Subnet objects having the ipam/SVC-k8s-lcm label defined. Therefore, all Subnet objects used for LCM networks must have the ipam/SVC-k8s-lcm label defined. For details, see Service labels and their life cycle.

    5. Set the addresses, gateway4, and nameservers fields of the management network bridge to fetch data from the kaas-mgmt subnet.

    6. Configure bonding options using the parameters field. The only mandatory option is mode. See the example below for details.

      Note

      You can set any mode supported by netplan and your hardware.

      Important

      Bond monitoring is disabled in Ubuntu by default. However, Mirantis highly recommends enabling it using Media Independent Interface (MII) monitoring by setting the mii-monitor-interval parameter to a non-zero value. For details, see Linux documentation: bond monitoring.

  2. Verify your configuration using the following example:

    kind: L2Template
    metadata:
      name: kaas-mgmt
      ...
    spec:
      ...
      l3Layout:
        - subnetName: kaas-mgmt
          scope:      namespace
      npTemplate: |
        version: 2
        ethernets:
          {{nic 0}}:
            dhcp4: false
            dhcp6: false
            match:
              macaddress: {{mac 0}}
            set-name: {{nic 0}}
          {{nic 1}}:
            dhcp4: false
            dhcp6: false
            match:
              macaddress: {{mac 1}}
            set-name: {{nic 1}}
        bonds:
          bond0:
            interfaces:
              - {{nic 0}}
              - {{nic 1}}
            parameters:
              mode: 802.3ad
              mii-monitor-interval: 100
            dhcp4: false
            dhcp6: false
        bridges:
          k8s-lcm:
            interfaces: [bond0]
            addresses:
              - {{ip "k8s-lcm:kaas-mgmt"}}
            gateway4: {{gateway_from_subnet "kaas-mgmt"}}
            nameservers:
              addresses: {{nameservers_from_subnet "kaas-mgmt"}}
        ...
    
  3. Proceed to bootstrap your management cluster as described in Bootstrap a management cluster.

Separate PXE and management networks

This section describes how to configure a dedicated PXE network for a management or regional bare metal cluster. A separate PXE network allows isolating sensitive bare metal provisioning process from the end users. The users still have access to Container Cloud services, such as Keycloak, to authenticate workloads in managed clusters, such as Horizon in a Mirantis OpenStack for Kubernetes cluster.

Note

This additional configuration procedure must be completed as part of the Prepare metadata and deploy the management cluster before 2.24.0 steps. It substitutes or appends some configuration parameters and templates that are used in Prepare metadata and deploy the management cluster before 2.24.0 for the management cluster to use two networks, PXE and management, instead of one PXE/management network. We recommend considering the Prepare metadata and deploy the management cluster before 2.24.0 procedure first.

The following table describes the overall network mapping scheme with all L2/L3 parameters, for example, for two networks, PXE (CIDR 10.0.0.0/24) and management (CIDR 10.0.11.0/24):

Network mapping overview

Deployment file name

Network

Parameters and values

cluster.yaml

Management

  • SET_LB_HOST=10.0.11.90

  • SET_METALLB_ADDR_POOL=10.0.11.61-10.0.11.80

ipam-objects.yaml

PXE

  • SET_IPAM_CIDR=10.0.0.0/24

  • SET_PXE_NW_GW=10.0.0.1

  • SET_PXE_NW_DNS=8.8.8.8

  • SET_IPAM_POOL_RANGE=10.0.0.100-10.0.0.109

  • SET_METALLB_PXE_ADDR_POOL=10.0.0.61-10.0.0.70

ipam-objects.yaml

Management

  • SET_LCM_CIDR=10.0.11.0/24

  • SET_LCM_RANGE=10.0.11.100-10.0.11.199

  • SET_LB_HOST=10.0.11.90

  • SET_METALLB_ADDR_POOL=10.0.11.61-10.0.11.80

bootstrap.sh

PXE

  • KAAS_BM_PXE_IP=10.0.0.20

  • KAAS_BM_PXE_MASK=24

  • KAAS_BM_PXE_BRIDGE=br0

  • KAAS_BM_BM_DHCP_RANGE=10.0.0.30,10.0.0.59,255.255.255.0

  • BOOTSTRAP_METALLB_ADDRESS_POOL=10.0.0.61-10.0.0.80


When using separate PXE and management networks, the management cluster services are exposed in different networks using two separate MetalLB address pools:

  • Services exposed through the PXE network are as follows:

    • Ironic API as a bare metal provisioning server

    • HTTP server that provides images for network boot and server provisioning

    • Caching server for accessing the Container Cloud artifacts deployed on hosts

  • Services exposed through the management network are all other Container Cloud services, such as Keycloak, web UI, and so on.

To configure separate PXE and management networks:

  1. Inspect guidelines to follow during configuration of the Subnet object as a MetalLB address pool as described MetalLB configuration guidelines for subnets.

  2. To ensure successful bootstrap, enable asymmetric routing on the interfaces of the management cluster nodes. This is required because the seed node relies on one network by default, which can potentially cause traffic asymmetry.

    In the kernelParameters section of bm/baremetalhostprofiles.yaml.template, set rp_filter to 2. This enables loose mode as defined in RFC3704.

    Example configuration of asymmetric routing
    ...
    kernelParameters:
      ...
      sysctl:
        # Enables the "Loose mode" for the "k8s-lcm" interface (management network)
        net.ipv4.conf.k8s-lcm.rp_filter: "2"
        # Enables the "Loose mode" for the "bond0" interface (PXE network)
        net.ipv4.conf.bond0.rp_filter: "2"
        ...
    

    Note

    More complicated solutions that are not described in this manual include getting rid of traffic asymmetry, for example:

    • Configure source routing on management cluster nodes.

    • Plug the seed node into the same networks as the management cluster nodes, which requires custom configuration of the seed node.

  3. In kaas-bootstrap/templates/bm/ipam-objects.yaml.template:

    • Substitute all the Subnet object templates with the new ones as described in the example template below

    • Update the L2 template spec.l3Layout and spec.npTemplate fields as described in the example template below

    Example of the Subnet object templates
    # Subnet object that provides IP addresses for bare metal hosts of
    # management cluster in the PXE network.
    apiVersion: "ipam.mirantis.com/v1alpha1"
    kind: Subnet
    metadata:
      name: mgmt-pxe
      namespace: default
      labels:
        kaas.mirantis.com/provider: baremetal
        kaas.mirantis.com/region: region-one
        kaas-mgmt-pxe-subnet: ""
    spec:
      cidr: SET_IPAM_CIDR
      gateway: SET_PXE_NW_GW
      nameservers:
        - SET_PXE_NW_DNS
      includeRanges:
        - SET_IPAM_POOL_RANGE
      excludeRanges:
        - SET_METALLB_PXE_ADDR_POOL
    ---
    # Subnet object that provides IP addresses for bare metal hosts of
    # management cluster in the management network.
    apiVersion: "ipam.mirantis.com/v1alpha1"
    kind: Subnet
    metadata:
      name: mgmt-lcm
      namespace: default
      labels:
        kaas.mirantis.com/provider: baremetal
        kaas.mirantis.com/region: region-one
        kaas-mgmt-lcm-subnet: ""
        ipam/SVC-k8s-lcm: "1"
        ipam/SVC-ceph-cluster: "1"
        ipam/SVC-ceph-public: "1"
        cluster.sigs.k8s.io/cluster-name: CLUSTER_NAME
    spec:
      cidr: {{ SET_LCM_CIDR }}
      includeRanges:
        - {{ SET_LCM_RANGE }}
      excludeRanges:
        - SET_LB_HOST
        - SET_METALLB_ADDR_POOL
    ---
    # Subnet object that provides configuration for "services-pxe" MetalLB
    # address pool that will be used to expose services LB endpoints in the
    # PXE network.
    apiVersion: "ipam.mirantis.com/v1alpha1"
    kind: Subnet
    metadata:
      name: mgmt-pxe-lb
      namespace: default
      labels:
        kaas.mirantis.com/provider: baremetal
        kaas.mirantis.com/region: region-one
        ipam/SVC-MetalLB: ""
        metallb/address-pool-name: services-pxe
        metallb/address-pool-protocol: layer2
        metallb/address-pool-auto-assign: "false"
        cluster.sigs.k8s.io/cluster-name: CLUSTER_NAME
    spec:
      cidr: SET_IPAM_CIDR
      includeRanges:
        - SET_METALLB_PXE_ADDR_POOL
    
    Example of the L2 template spec
    kind: L2Template
    ...
    spec:
      ...
      l3Layout:
        - scope: namespace
          subnetName: kaas-mgmt-pxe
          labelSelector:
            kaas.mirantis.com/provider: baremetal
            kaas-mgmt-pxe-subnet: ""
        - scope: namespace
          subnetName: kaas-mgmt-lcm
          labelSelector:
            kaas.mirantis.com/provider: baremetal
            kaas-mgmt-lcm-subnet: ""
      npTemplate: |
        version: 2
        renderer: networkd
        ethernets:
          {{nic 0}}:
            dhcp4: false
            dhcp6: false
            match:
              macaddress: {{mac 0}}
            set-name: {{nic 0}}
          {{nic 1}}:
            dhcp4: false
            dhcp6: false
            match:
              macaddress: {{mac 1}}
            set-name: {{nic 1}}
        bridges:
          bm-pxe:
            interfaces:
             - {{ nic 0 }}
            dhcp4: false
            dhcp6: false
            addresses:
              - {{ ip "bm-pxe:kaas-mgmt-pxe" }}
            nameservers:
              addresses: {{ nameservers_from_subnet "kaas-mgmt-pxe" }}
            routes:
              - to: 0.0.0.0/0
                via: {{ gateway_from_subnet "kaas-mgmt-pxe" }}
          k8s-lcm:
            interfaces:
             - {{ nic 1 }}
            dhcp4: false
            dhcp6: false
            addresses:
              - {{ ip "k8s-lcm:kaas-mgmt-lcm" }}
            nameservers:
              addresses: {{ nameservers_from_subnet "kaas-mgmt-lcm" }}
    

    The last Subnet template named mgmt-pxe-lb in the example above will be used to configure the MetalLB address pool in the PXE network. The bare metal provider will automatically configure MetalLB with address pools using the Subnet objects identified by specific labels.

    Warning