The documentation is intended to help operators understand the core concepts
of the product.
The information provided in this documentation set is being constantly
improved and amended based on the feedback and kind requests from our
software consumers. This documentation set outlines description of
the features supported within three latest Container Cloud minor releases and
their supported Cluster releases, with a corresponding note
Available since <release-version>.
The following table lists the guides included in the documentation set you
are reading:
GUI elements that include any part of interactive user interface and
menu navigation.
Superscript
Some extra, brief information. For example, if a feature is
available from a specific release or if a feature is in the
Technology Preview development stage.
Note
The Note block
Messages of a generic meaning that may be useful to the user.
Caution
The Caution block
Information that prevents a user from mistakes and undesirable
consequences when following the procedures.
Warning
The Warning block
Messages that include details that can be easily missed, but should not
be ignored by the user and are valuable before proceeding.
See also
The See also block
List of references that may be helpful for understanding of some related
tools, concepts, and so on.
Learn more
The Learn more block
Used in the Release Notes to wrap a list of internal references to
the reference architecture, deployment and operation procedures specific
to a newly implemented product feature.
A Technology Preview feature provides early access to upcoming product
innovations, allowing customers to experiment with the functionality and
provide feedback.
Technology Preview features may be privately or publicly available but
neither are intended for production use. While Mirantis will provide
assistance with such features through official channels, normal Service
Level Agreements do not apply.
As Mirantis considers making future iterations of Technology Preview features
generally available, we will do our best to resolve any issues that customers
experience when using these features.
During the development of a Technology Preview feature, additional components
may become available to the public for evaluation. Mirantis cannot guarantee
the stability of such features. As a result, if you are using Technology
Preview features, you may not be able to seamlessly update to subsequent
product releases, as well as upgrade or migrate to the functionality that
has not been announced as full support yet.
Mirantis makes no guarantees that Technology Preview features will graduate
to generally available features.
The documentation set refers to Mirantis Container Cloud GA as to the latest
released GA version of the product. For details about the Container Cloud
GA minor releases dates, refer to
Container Cloud releases.
Mirantis Container Cloud enables you to ship code faster by enabling
speed with choice, simplicity, and security. Through a single pane
of glass you can deploy, manage, and observe Kubernetes
clusters on private clouds or bare metal infrastructure.
Mirantis Container Cloud provides the ability to leverage the following on
premises cloud infrastructure: OpenStack, VMware, and bare metal.
The list of the most common use cases includes:
Multi-cloud
Organizations are increasingly moving toward a multi-cloud strategy,
with the goal of enabling the effective placement of workloads over
multiple platform providers. Multi-cloud strategies can introduce
a lot of complexity and management overhead. Mirantis Container Cloud
enables you to effectively deploy and manage container clusters
(Kubernetes and Swarm) across multiple cloud provider platforms.
Hybrid cloud
The challenges of consistently deploying, tracking, and managing hybrid
workloads across multiple cloud platforms is compounded by not having
a single point that provides information on all available resources.
Mirantis Container Cloud enables hybrid cloud workload by providing
a central point of management and visibility of all your cloud resources.
Kubernetes cluster lifecycle management
The consistent lifecycle management of a single Kubernetes cluster
is a complex task on its own that is made infinitely more difficult
when you have to manage multiple clusters across different platforms
spread across the globe. Mirantis Container Cloud provides a single,
centralized point from which you can perform full lifecycle management
of your container clusters, including automated updates and upgrades.
We also support attaching existing Mirantis Kubernetes Engine clusters.
Highly regulated industries
Regulated industries need a fine level of access control granularity,
high security standards and extensive reporting capabilities to ensure
that they can meet and exceed the security standards and requirements.
Mirantis Container Cloud provides for a fine-grained Role Based Access
Control (RBAC) mechanism and easy integration and federation to existing
identity management systems (IDM).
Logging, monitoring, alerting
A complete operational visibility is required to identify and address issues
in the shortest amount of time – before the problem becomes serious.
Mirantis StackLight is the proactive monitoring, logging, and alerting
solution designed for large-scale container and cloud observability with
extensive collectors, dashboards, trend reporting and alerts.
Storage
Cloud environments require a unified pool of storage that can be scaled up by
simply adding storage server nodes. Ceph is a unified, distributed storage
system designed for excellent performance, reliability, and scalability.
Deploy Ceph utilizing Rook to provide and manage a robust persistent storage
that can be used by Kubernetes workloads on the baremetal-based clusters.
Security
Security is a core concern for all enterprises, especially with more
of our systems being exposed to the Internet as a norm. Mirantis
Container Cloud provides for a multi-layered security approach that
includes effective identity management and role based authentication,
secure out of the box defaults and extensive security scanning and
monitoring during the development process.
5G and Edge
The introduction of 5G technologies and the support of Edge workloads
requires an effective multi-tenant solution to manage the underlying
container infrastructure. Mirantis Container Cloud provides for a full
stack, secure, multi-cloud cluster management and Day-2 operations
solution that supports both on premises bare metal and cloud.
Mirantis Container Cloud is a set of microservices
that are deployed using Helm charts and run in a Kubernetes cluster.
Container Cloud is based on the Kubernetes Cluster API community initiative.
The following diagram illustrates an overview of Container Cloud
and the clusters it manages:
All artifacts used by Kubernetes and workloads are stored
on the Container Cloud content delivery network (CDN):
mirror.mirantis.com (Debian packages including the Ubuntu mirrors)
binary.mirantis.com (Helm charts and binary artifacts)
mirantis.azurecr.io (Docker image registry)
All Container Cloud components are deployed in the Kubernetes clusters.
All Container Cloud APIs are implemented using the Kubernetes
Custom Resource Definition (CRD) that represents custom objects
stored in Kubernetes and allows you to expand Kubernetes API.
The Container Cloud logic is implemented using controllers.
A controller handles the changes in custom resources defined
in the controller CRD.
A custom resource consists of a spec that describes the desired state
of a resource provided by a user.
During every change, a controller reconciles the external state of a custom
resource with the user parameters and stores this external state in the
status subresource of its custom resource.
Container Cloud can have several regions. A region is a physical location,
for example, a data center, that has access to one or several cloud provider
back ends. A separate regional cluster manages a region that can include
multiple providers. A region must have a two-way (full) network connectivity
between a regional cluster and a cloud provider back end. For example,
an OpenStack VM must have access to the related regional cluster. And this
regional cluster must have access to the OpenStack floating IPs and
load balancers.
The following diagram illustrates the structure of the Container Cloud regions:
The types of the Container Cloud clusters include:
Bootstrap cluster
Runs the bootstrap process on a seed node. For the OpenStack or VMware
vSphere-based Container Cloud, it can be an operator desktop computer.
For the baremetal-based Container Cloud, this is the first temporary
data center node.
Requires access to one of the following provider back ends: OpenStack,
vSphere, or bare metal.
Contains minimum set of services to deploy
the management and regional clusters.
Is destroyed completely after a successful bootstrap.
Management and regional clusters
Management cluster:
Runs all public APIs and services including the web UIs
of Container Cloud.
Does not require access to any provider back end.
Regional cluster:
Is combined with management cluster by default.
Runs the provider-specific services and internal API including
LCMMachine and LCMCluster. Also, it runs an LCM controller for
orchestrating managed clusters and other controllers for handling
different resources.
Requires two-way access to a provider back end. The provider connects
to a back end to spawn managed cluster nodes,
and the agent running on the nodes accesses the regional cluster
to obtain the deployment information.
Requires access to a management cluster to obtain user parameters.
Supports multi-regional deployments.
For example, you can deploy a vSphere-based management cluster
with vSphere-based and OpenStack-based regional clusters.
A Mirantis Kubernetes Engine (MKE) cluster that an end user
creates using the Container Cloud web UI.
Requires access to a regional cluster. Each node of a managed
cluster runs an LCM Agent that connects to the LCM machine of the
regional cluster to obtain the deployment details.
An attached MKE cluster that is not created using
Container Cloud. In such case, nodes of the attached cluster
do not contain LCM Agent. For supported MKE versions that can be attached
to Container Cloud, see Release Compatibility Matrix.
Baremetal-based managed clusters support the Mirantis OpenStack for Kubernetes
(MOSK) product. For details, see
MOSK documentation.
All types of the Container Cloud clusters except the bootstrap cluster
are based on the MKE and Mirantis Container Runtime (MCR) architecture.
For details, see MKE and
MCR documentation.
The following diagram illustrates the distribution of services
between each type of the Container Cloud clusters:
The Mirantis Container Cloud provider is the central component
of Container Cloud that provisions a node of a management, regional,
or managed cluster and runs the LCM Agent on this node.
It runs in a management and regional clusters and requires connection
to a provider back end.
The Container Cloud provider interacts with the following
types of public API objects:
Public API object name
Description
Container Cloud release object
Contains the following information about clusters:
Version of the supported Cluster release for a management and
regional clusters
List of supported Cluster releases for the managed clusters
and supported upgrade path
Description of Helm charts that are installed
on the management and regional clusters
depending on the selected provider
Cluster release object
Provides a specific version of a management, regional, or
managed cluster.
Any Cluster release object, as well as a Container Cloud release
object never changes, only new releases can be added.
Any change leads to a new release of a cluster.
Contains references to all components and their versions
that are used to deploy all cluster types:
LCM components:
LCM Agent
Ansible playbooks
Scripts
Description of steps to execute during a cluster deployment
and upgrade
Helm Controller image references
Supported Helm charts description:
Helm chart name and version
Helm release name
Helm values
Cluster object
References the Credentials, KaaSRelease and ClusterRelease
objects.
Is tied to a specific Container Cloud region and provider.
Represents all cluster-level resources. For example, for the
OpenStack-based clusters, it represents networks, load balancer for
the Kubernetes API, and so on. It uses data from the Credentials
object to create these resources and data from the KaaSRelease
and ClusterRelease objects to ensure that all lower-level cluster
objects are created.
Machine object
References the Cluster object.
Represents one node of a managed cluster, for example, an OpenStack VM,
and contains all data to provision it.
Credentials object
Contains all information necessary to connect to a provider back end.
Is tied to a specific Container Cloud region and provider.
PublicKey object
Is provided to every machine to obtain an SSH access.
The following diagram illustrates the Container Cloud provider data flow:
The Container Cloud provider performs the following operations
in Container Cloud:
Consumes the below types of data from a management and regional cluster:
Credentials to connect to a provider back end
Deployment instructions from the KaaSRelease and ClusterRelease
objects
The cluster-level parameters from the Cluster objects
The machine-level parameters from the Machine objects
Prepares data for all Container Cloud components:
Creates the LCMCluster and LCMMachine custom resources
for LCM Controller and LCM Agent. The LCMMachine custom resources
are created empty to be later handled by the LCM Controller.
Creates the HelmBundle custom resources for the Helm Controller
using data from the KaaSRelease and ClusterRelease objects.
Creates service accounts for these custom resources.
Creates a scope in Identity and access management (IAM)
for a user access to a managed cluster.
Provisions nodes for a managed cluster using the cloud-init script
that downloads and runs the LCM Agent.
The Mirantis Container Cloud Release Controller is responsible
for the following functionality:
Monitor and control the KaaSRelease and ClusterRelease objects
present in a management cluster. If any release object is used
in a cluster, the Release Controller prevents the deletion
of such an object.
Trigger the Container Cloud auto-upgrade procedure if a new
KaaSRelease object is found:
Search for the managed clusters with old Cluster releases
that are not supported by a new Container Cloud release.
If any are detected, abort the auto-upgrade and display
a corresponding note about an old Cluster release in the Container
Cloud web UI for the managed clusters. In this case, a user must update
all managed clusters using the Container Cloud web UI.
Once all managed clusters are upgraded to the Cluster releases
supported by a new Container Cloud release,
the Container Cloud auto-upgrade is retriggered
by the Release Controller.
Trigger the Container Cloud release upgrade of all Container Cloud
components in a management cluster.
The upgrade itself is processed by the Container Cloud provider.
Trigger the Cluster release upgrade of a management cluster
to the Cluster release version that is indicated
in the upgraded Container Cloud release version.
The LCMCluster components, such as MKE, are upgraded before
the HelmBundle components, such as StackLight or Ceph.
Verify the regional cluster(s) status. If the regional cluster
is ready, trigger the Cluster release upgrade
of the regional cluster.
Once a management cluster is upgraded, an option to update
a managed cluster becomes available in the Container Cloud web UI.
During a managed cluster update, all cluster components including
Kubernetes are automatically upgraded to newer versions if available.
The LCMCluster components, such as MKE, are upgraded before
the HelmBundle components, such as StackLight or Ceph.
The Operator can delay the Container Cloud automatic upgrade procedure for a
limited amount of time or schedule upgrade to run at desired hours or weekdays.
For details, see Schedule Mirantis Container Cloud upgrades.
Container Cloud remains operational during the management and
regional clusters upgrade. Managed clusters are not affected
during this upgrade. For the list of components that are updated during
the Container Cloud upgrade, see the Components versions section
of the corresponding Container Cloud release in
Release Notes.
When Mirantis announces support of the newest versions of
Mirantis Container Runtime (MCR) and Mirantis Kubernetes Engine
(MKE), Container Cloud automatically upgrades these components as well.
For the maintenance window best practices before upgrade of these
components, see
MKE Documentation.
The Mirantis Container Cloud web UI is mainly designed
to create and update the managed clusters as well as add or remove machines
to or from an existing managed cluster.
It also allows attaching existing Mirantis Kubernetes Engine (MKE) clusters.
You can use the Container Cloud web UI
to obtain the management cluster details including endpoints, release version,
and so on.
The management cluster update occurs automatically
with a new release change log available through the Container Cloud web UI.
The Container Cloud web UI is a JavaScript application that is based
on the React framework. The Container Cloud web UI is designed to work
on a client side only. Therefore, it does not require a special back end.
It interacts with the Kubernetes and Keycloak APIs directly.
The Container Cloud web UI uses a Keycloak token
to interact with Container Cloud API and download kubeconfig
for the management and managed clusters.
The Container Cloud web UI uses NGINX that runs on a management cluster
and handles the Container Cloud web UI static files.
NGINX proxies the Kubernetes and Keycloak APIs
for the Container Cloud web UI.
The bare metal service provides for the discovery, deployment, and management
of bare metal hosts.
The bare metal management in Mirantis Container Cloud
is implemented as a set of modular microservices.
Each microservice implements a certain requirement or function
within the bare metal management system.
The back-end bare metal manager in a standalone mode with its auxiliary
services that include httpd, dnsmasq, and mariadb.
OpenStack Ironic Inspector
Introspects and discovers the bare metal hosts inventory.
Includes OpenStack Ironic Python Agent (IPA) that is used
as a provision-time agent for managing bare metal hosts.
Ironic Operator
Monitors changes in the external IP addresses of httpd, ironic,
and ironic-inspector and automatically reconciles the configuration
for dnsmasq, ironic, baremetal-provider,
and baremetal-operator.
Bare Metal Operator
Manages bare metal hosts through the Ironic API. The Container Cloud
bare-metal operator implementation is based on the Metal³ project.
Bare metal resources manager
Ensures that the bare metal provisioning artifacts such as the
distribution image of the operating system is available and up to date.
cluster-api-provider-baremetal
The plugin for the Kubernetes Cluster API integrated with Container Cloud.
Container Cloud uses the Metal³ implementation of
cluster-api-provider-baremetal for the Cluster API.
HAProxy
Load balancer for external access to the Kubernetes API endpoint.
LCM Agent
Used for physical and logical storage, physical and logical network,
and control over the life cycle of a bare metal machine resources.
Ceph
Distributed shared storage is required by the Container Cloud services
to create persistent volumes to store their data.
MetalLB
Load balancer for Kubernetes services on bare metal. 1
Keepalived
Monitoring service that ensures availability of the virtual IP for
the external load balancer endpoint (HAProxy). 1
IPAM
IP address management services provide consistent IP address space
to the machines in bare metal clusters. See details in
IP Address Management.
Mirantis Container Cloud on bare metal uses IP Address Management (IPAM)
to keep track of the network addresses allocated to bare metal hosts.
This is necessary to avoid IP address conflicts
and expiration of address leases to machines through DHCP.
Note
Only IPv4 address family is currently supported by Container Cloud
and IPAM. IPv6 is not supported and not used in Container Cloud.
IPAM is provided by the kaas-ipam controller. Its functions
include:
Allocation of IP address ranges or subnets to newly created clusters using
SubnetPool and Subnet resources.
Allocation IP addresses to machines and cluster services at the request
of baremetal-provider using the IpamHost and IPaddr resources.
Creation and maintenance of host networking configuration
on the bare metal hosts using the IpamHost resources.
The IPAM service can support different networking topologies and network
hardware configurations on the bare metal hosts.
In the most basic network configuration, IPAM uses a single L3 network
to assign addresses to all bare metal hosts, as defined in
Managed cluster networking.
You can apply complex networking configurations to a bare metal host
using the L2 templates. The L2 templates imply multihomed host networking
and enable you to create a managed cluster where nodes use separate host
networks for different types of traffic. Multihoming is required
to ensure the security and performance of a managed cluster.
Caution
Modification of L2 templates in use is allowed with a mandatory
validation step from the Infrastructure Operator to prevent accidental
cluster failures due to unsafe changes. The list of risks posed by modifying
L2 templates includes:
Services running on hosts cannot reconfigure automatically to switch to
the new IP addresses and/or interfaces.
Connections between services are interrupted unexpectedly, which can cause
data loss.
Incorrect configurations on hosts can lead to irrevocable loss of
connectivity between services and unexpected cluster partition or
disassembly.
The main purpose of networking in a Container Cloud management or regional
cluster is to provide access to the Container Cloud Management API
that consists of the Kubernetes API of the Container Cloud management and
regional clusters and the Container Cloud LCM API. This API allows end users to
provision and configure managed clusters and machines. Also, this API is used
by LCM Agents in managed clusters to obtain configuration and report status.
The following types of networks are supported for the management and regional
clusters in Container Cloud:
PXE network
Enables PXE boot of all bare metal machines in the Container Cloud region.
PXE subnet
Provides IP addresses for DHCP and network boot of the bare metal hosts
for initial inspection and operating system provisioning.
This network may not have the default gateway or a router connected
to it. The PXE subnet is defined by the Container Cloud Operator
during bootstrap.
Provides IP addresses for the bare metal management services of
Container Cloud, such as bare metal provisioning service (Ironic).
These addresses are allocated and served by MetalLB.
Management network
Connects LCM Agents running on the hosts to the Container Cloud LCM API.
Serves the external connections to the Container Cloud Management API.
The network is also used for communication between kubelet
and the Kubernetes API server inside a Kubernetes cluster. The MKE
components use this network for communication inside a swarm cluster.
LCM subnet
Provides IP addresses for the Kubernetes nodes in the management cluster.
This network also provides a Virtual IP (VIP) address for the load
balancer that enables external access to the Kubernetes API
of a management cluster. This VIP is also the endpoint to access
the Container Cloud Management API in the management cluster.
Provides IP addresses for the externally accessible services of
Container Cloud, such as Keycloak, web UI, StackLight.
These addresses are allocated and served by MetalLB.
Kubernetes workloads network
Technology Preview
Serves the internal traffic between workloads on the management cluster.
Kubernetes workloads subnet
Provides IP addresses that are assigned to nodes and used by Calico.
Out-of-Band (OOB) network
Connects to Baseboard Management Controllers of the servers that host
the management cluster. The OOB subnet must be accessible from the
management network through IP routing. The OOB network
is not managed by Container Cloud and is not represented in the IPAM API.
A Kubernetes cluster networking is typically focused on connecting pods on
different nodes. On bare metal, however, the cluster networking is more
complex as it needs to facilitate many different types of traffic.
Kubernetes clusters managed by Mirantis Container Cloud
have the following types of traffic:
PXE network
Enables the PXE boot of all bare metal machines in Container Cloud.
This network is not configured on the hosts in a managed cluster.
It is used by the bare metal provider to provision additional
hosts in managed clusters and is disabled on the hosts after
provisioning is done.
Life-cycle management (LCM) network
Connects LCM Agents running on the hosts to the Container Cloud LCM API.
The LCM API is provided by the regional or management cluster.
The LCM network is also used for communication between kubelet
and the Kubernetes API server inside a Kubernetes cluster. The MKE
components use this network for communication inside a swarm cluster.
LCM subnet
Provides IP addresses that are statically allocated by the IPAM service
to bare metal hosts. This network must be connected to the Kubernetes API
endpoint of the regional cluster through an IP router.
LCM Agents running on managed clusters will connect to
the regional cluster API through this router. LCM subnets may be
different per managed cluster as long as this connection requirement is
satisfied.
The Virtual IP (VIP) address for load balancer that enables access to
the Kubernetes API of the managed cluster must be allocated from the LCM
subnet.
Kubernetes workloads network
Technology Preview
Serves as an underlay network for traffic between pods in
the managed cluster. This network should not be shared between clusters.
Kubernetes workloads subnet
Provides IP addresses that are assigned to nodes and used by Calico.
Kubernetes external network
Serves ingress traffic to the managed cluster from the outside world.
This network can be shared between clusters, but must have a dedicated
subnet per cluster.
Services subnet
Technology Preview
Provides IP addresses for externally available load-balanced services.
The address ranges for MetalLB are assigned from this subnet.
This subnet must be unique per managed cluster.
Storage network
Serves storage access and replication traffic from and to Ceph OSD services.
The storage network does not need to be connected to any IP routers
and does not require external access, unless you want to use Ceph
from outside of a Kubernetes cluster.
To use a dedicated storage network, define and configure
both subnets listed below.
Storage access subnet
Provides IP addresses that are assigned to Ceph nodes.
The Ceph OSD services bind to these addresses on their respective
nodes. Serves Ceph access traffic from and to storage clients.
This is a public network in Ceph terms. 1
This subnet is unique per managed cluster.
Storage replication subnet
Provides IP addresses that are assigned to Ceph nodes.
The Ceph OSD services bind to these addresses on their respective
nodes. Serves Ceph internal replication traffic. This is a
cluster network in Ceph terms. 1
This subnet is unique per managed cluster.
Out-of-Band (OOB) network
Connects baseboard management controllers (BMCs) of the bare metal hosts.
This network must not be accessible from the managed clusters.
The following diagram illustrates the networking schema of the Container Cloud
deployment on bare metal with a managed cluster:
The following network roles are defined for all Mirantis Container Cloud
clusters nodes on bare metal including the bootstrap,
management, regional, and managed cluster nodes:
Out-of-band (OOB) network
Connects the Baseboard Management Controllers (BMCs) of the hosts
in the network to Ironic. This network is out of band for the
host operating system.
PXE network
Enables remote booting of servers through the PXE protocol. In management
or regional clusters, DHCP server listens on this network for hosts
discovery and inspection. In managed clusters, hosts use this network
for the initial PXE boot and provisioning.
LCM network
Connects LCM Agents running on the node to the LCM API of the management
or regional cluster. It is also used for communication between kubelet
and the Kubernetes API server inside a Kubernetes cluster. The MKE
components use this network for communication inside a swarm cluster.
In management or regional clusters, it is replaced by the management
network.
Kubernetes workloads (pods) network
Technology Preview
Serves connections between Kubernetes pods.
Each host has an address on this network, and this address is used
by Calico as an endpoint to the underlay network.
Kubernetes external network
Technology Preview
Serves external connection to the Kubernetes API
and the user services exposed by the cluster. In management or regional
clusters, it is replaced by the management network.
Management network
Serves external connections to the Container Cloud Management API and
services of the management or regional cluster.
Not available in a managed cluster.
Storage access network
Connects Ceph nodes to the storage clients. The Ceph OSD service is
bound to the address on this network. This is a public network in
Ceph terms. 0
Storage replication network
Connects Ceph nodes to each other. Serves internal replication traffic.
This is a cluster network in Ceph terms. 0
Each network is represented on the host by a virtual Linux bridge. Physical
interfaces may be connected to one of the bridges directly, or through a
logical VLAN subinterface, or combined into a bond interface that is in
turn connected to a bridge.
The following table summarizes the default names used for the bridges
connected to the networks listed above:
The baremetal-based Mirantis Container Cloud uses Ceph as a distributed
storage system for file, block, and object storage. This section provides an
overview of a Ceph cluster deployed by Container Cloud.
Mirantis Container Cloud deploys Ceph on baremetal-based managed clusters
using Helm charts with the following components:
Rook Ceph Operator
A storage orchestrator that deploys Ceph on top of a Kubernetes cluster. Also
known as Rook or RookOperator. Rook operations include:
Deploying and managing a Ceph cluster based on provided Rook CRs such as
CephCluster, CephBlockPool, CephObjectStore, and so on.
Orchestrating the state of the Ceph cluster and all its daemons.
KaaSCephCluster custom resource (CR)
Represents the customization of a Kubernetes installation and allows you to
define the required Ceph configuration through the Container Cloud web UI
before deployment. For example, you can define the failure domain, Ceph pools,
Ceph node roles, number of Ceph components such as Ceph OSDs, and so on.
The ceph-kcc-controller controller on the Container Cloud management
cluster manages the KaaSCephCluster CR.
Ceph Controller
A Kubernetes controller that obtains the parameters from Container Cloud
through a CR, creates CRs for Rook and updates its CR status based on the Ceph
cluster deployment progress. It creates users, pools, and keys for OpenStack
and Kubernetes and provides Ceph configurations and keys to access them. Also,
Ceph Controller eventually obtains the data from the OpenStack Controller for
the Keystone integration and updates the RADOS Gateway services configurations
to use Kubernetes for user authentication. Ceph Controller operations include:
Transforming user parameters from the Container Cloud Ceph CR into Rook CRs
and deploying a Ceph cluster using Rook.
Providing integration of the Ceph cluster with Kubernetes.
Providing data for OpenStack to integrate with the deployed Ceph cluster.
Ceph Status Controller
A Kubernetes controller that collects all valuable parameters from the current
Ceph cluster, its daemons, and entities and exposes them into the
KaaSCephCluster status. Ceph Status Controller operations include:
Collecting all statuses from a Ceph cluster and corresponding Rook CRs.
Collecting additional information on the health of Ceph daemons.
Provides information to the status section of the KaaSCephCluster
CR.
Ceph Request Controller
A Kubernetes controller that obtains the parameters from Container Cloud
through a CR and manages Ceph OSD lifecycle management (LCM) operations. It
allows for a safe Ceph OSD removal from the Ceph cluster. Ceph Request
Controller operations include:
Providing an ability to perform Ceph OSD LCM operations.
Obtaining specific CRs to remove Ceph OSDs and executing them.
Pausing the regular Ceph Controller reconcile until all requests are
completed.
A typical Ceph cluster consists of the following components:
Ceph Monitors - three or, in rare cases, five Ceph Monitors.
Ceph Managers:
Before Container Cloud 2.22.0, one Ceph Manager.
Since Container Cloud 2.22.0, two Ceph Managers.
RADOS Gateway services - Mirantis recommends having three or more RADOS
Gateway instances for HA.
Ceph OSDs - the number of Ceph OSDs may vary according to the deployment
needs.
Warning
A Ceph cluster with 3 Ceph nodes does not provide
hardware fault tolerance and is not eligible
for recovery operations,
such as a disk or an entire Ceph node replacement.
A Ceph cluster uses the replication factor that equals 3.
If the number of Ceph OSDs is less than 3, a Ceph cluster
moves to the degraded state with the write operations
restriction until the number of alive Ceph OSDs
equals the replication factor again.
The placement of Ceph Monitors and Ceph Managers is defined in the
KaaSCephCluster CR.
The following diagram illustrates the way a Ceph cluster is deployed in
Container Cloud:
The following diagram illustrates the processes within a deployed Ceph cluster:
A Ceph cluster configuration in Mirantis Container Cloud
includes but is not limited to the following limitations:
Only one Ceph Controller per a managed cluster and only one Ceph cluster per
Ceph Controller are supported.
The replication size for any Ceph pool must be set to more than 1.
Only one CRUSH tree per cluster. The separation of devices per Ceph pool is
supported through device classes
with only one pool of each type for a device class.
All CRUSH rules must have the same failure_domain.
Only the following types of CRUSH buckets are supported:
topology.kubernetes.io/region
topology.kubernetes.io/zone
topology.rook.io/datacenter
topology.rook.io/room
topology.rook.io/pod
topology.rook.io/pdu
topology.rook.io/row
topology.rook.io/rack
topology.rook.io/chassis
Consuming an existing Ceph cluster is not supported.
CephFS is not fully supported TechPreview.
Only IPv4 is supported.
If two or more Ceph OSDs are located on the same device, there must be no
dedicated WAL or DB for this class.
Only a full collocation or dedicated WAL and DB configurations are supported.
The minimum size of any defined Ceph OSD device is 5 GB.
Reducing the number of Ceph Monitors is not supported and causes the Ceph
Monitor daemons removal from random nodes.
Removal of the mgr role in the nodes section of the
KaaSCephCluster CR does not remove Ceph Managers. To remove a Ceph
Manager from a node, remove it from the nodes spec and manually delete
the mgr pod in the Rook namespace.
When adding a Ceph node with the Ceph Monitor role, if any issues occur with
the Ceph Monitor, rook-ceph removes it and adds a new Ceph Monitor instead,
named using the next alphabetic character in order. Therefore, the Ceph Monitor
names may not follow the alphabetical order. For example, a, b, d,
instead of a, b, c.
Ceph cluster does not support removable devices (with hotplug enabled) for
deploying Ceph OSDs.
Mirantis Container Cloud provides APIs that enable you
to define hardware configurations that extend the reference architecture:
Bare Metal Host Profile API
Enables for quick configuration of host boot and storage devices
and assigning of custom configuration profiles to individual machines.
See Create a custom bare metal host profile.
IP Address Management API
Enables for quick configuration of host network interfaces and IP addresses
and setting up of IP addresses ranges for automatic allocation.
See Create L2 templates.
Typically, operations with the extended hardware configurations are available
through the API and CLI, but not the web UI.
To keep operating system on a bare metal host up to date with the latest
security updates, the operating system requires periodic software
packages upgrade that may or may not require the host reboot.
Mirantis Container Cloud uses life cycle management tools to update
the operating system packages on the bare metal hosts. Container Cloud
may also trigger restart of bare metal hosts to apply the updates.
In the management cluster of Container Cloud, software package upgrade and
host restart is applied automatically when a new Container Cloud version
with available kernel or software packages upgrade is released.
In managed clusters, package upgrade and host restart is applied
as part of usual cluster upgrade using the Update cluster option
in the Container Cloud web UI.
Operating system upgrade and host restart are applied to cluster
nodes one by one. If Ceph is installed in the cluster, the Container
Cloud orchestration securely pauses the Ceph OSDs on the node before
restart. This allows avoiding degradation of the storage service.
Caution
Depending on the cluster configuration, applying security
updates and host restart can increase the update time for each node to up to
1 hour.
Cluster nodes are updated one by one. Therefore, for large clusters,
the update may take several days to complete.
The Mirantis Container Cloud managed clusters that are based on vSphere or
bare metal use MetalLB for load balancing of services and HAProxy with VIP
managed by Virtual Router Redundancy Protocol (VRRP) with Keepalived for the
Kubernetes API load balancer.
Every control plane node of each Kubernetes cluster runs the kube-api
service in a container. This service provides a Kubernetes API endpoint.
Every control plane node also runs the haproxy server that provides
load balancing with back-end health checking for all kube-api endpoints as
back ends.
The default load balancing method is least_conn. With this method,
a request is sent to the server with the least number of active
connections. The default load balancing method cannot be changed
using the Container Cloud API.
Only one of the control plane nodes at any given time serves as a
front end for Kubernetes API. To ensure this, the Kubernetes clients
use a virtual IP (VIP) address for accessing Kubernetes API.
This VIP is assigned to one node at a time using VRRP. Keepalived running on
each control plane node provides health checking and failover of the VIP.
Keepalived is configured in multicast mode.
Note
The use of VIP address for load balancing of Kubernetes API requires
that all control plane nodes of a Kubernetes cluster are connected
to a shared L2 segment. This limitation prevents from installing
full L3 topologies where control plane nodes are split
between different L2 segments and L3 networks.
Caution
External load balancers for services are not supported by
the current version of the Container Cloud vSphere provider.
The built-in load balancing described in this section is the only supported
option and cannot be disabled.
The services provided by the Kubernetes clusters, including
Container Cloud and user services, are balanced by MetalLB.
The metallb-speaker service runs on every worker node in
the cluster and handles connections to the service IP addresses.
MetalLB runs in the MAC-based (L2) mode. It means that all
control plane nodes must be connected to a shared L2 segment.
This is a limitation that does not allow installing full L3
cluster topologies.
Caution
External load balancers for services are not supported by
the current version of the Container Cloud vSphere provider.
The built-in load balancing described in this section is the only supported
option and cannot be disabled.
VMware vSphere network objects and IPAM recommendations¶
The VMware vSphere provider of Mirantis Container Cloud supports
the following types of vSphere network objects:
Virtual network
A network of virtual machines running on a hypervisor(s) that are logically
connected to each other so that they can exchange data. Virtual machines
can be connected to virtual networks that you create when you add a network.
Distributed port group
A port group associated with a vSphere distributed switch that specifies
port configuration options for each member port. Distributed port groups
define how connection is established through the vSphere distributed switch
to the network.
A Container Cloud cluster can be deployed using one of these network objects
with or without a DHCP server in the network:
Non-DHCP
Container Cloud uses IPAM service to manage IP addresses assignment to
machines. You must provide additional network parameters, such as
CIDR, gateway, IP ranges, and nameservers.
Container Cloud processes this data to the cloud-init metadata and
passes the data to machines during their bootstrap.
DHCP
Container Cloud relies on a DHCP server to assign IP addresses
to virtual machines.
Mirantis recommends using IP address management (IPAM) for cluster
machines provided by Container Cloud. IPAM must be enabled
for deployment in the non-DHCP vSphere networks. But Mirantis
recommends enabling IPAM in the DHCP-based networks as well. In this case,
the dedicated IPAM range should not intersect with the IP range used in the
DHCP server configuration for the provided vSphere network.
Such configuration prevents issues with accidental IP address change
for machines. For the issue details, see
vSphere troubleshooting.
Note
To obtain IPAM parameters for the selected vSphere network, contact
your vSphere administrator who provides you with IP ranges dedicated to your
environment only.
The following parameters are required to enable IPAM:
Network CIDR.
Network gateway address.
Minimum 1 DNS server.
IP address include range to be allocated for cluster machines.
Make sure that this range is not part of the DHCP range if the network has
a DHCP server.
Minimal number of addresses in the range:
3 IPs for management or regional cluster
3+N IPs for a managed cluster, where N is the number of worker nodes
Optional. IP address exclude range that is the list of IPs not to be
assigned to machines from the include ranges.
A dedicated Container Cloud network must not contain any virtual machines
with the keepalived instance running inside them as this may lead to the
vrouter_id conflict. By default, the Container Cloud management or
regional cluster is deployed with vrouter_id set to 1.
Managed clusters are deployed with the vrouter_id value starting from
2 and upper.
The Kubernetes lifecycle management (LCM) engine in Mirantis Container Cloud
consists of the following components:
LCM Controller
Responsible for all LCM operations. Consumes the LCMCluster object
and orchestrates actions through LCM Agent.
LCM Agent
Relates only to Mirantis Kubernetes Engine (MKE) clusters deployed
using Container Cloud, and is not used for attached MKE clusters.
Runs on the target host. Executes Ansible playbooks in headless mode.
Helm Controller
Responsible for the Helm charts life cycle, is installed by a cloud provider
as a Helm v3 chart.
The Kubernetes LCM components handle the following custom resources:
LCMCluster
LCMMachine
HelmBundle
The following diagram illustrates handling of the LCM custom resources by
the Kubernetes LCM components. On a managed cluster,
apiserver handles multiple Kubernetes objects,
for example, deployments, nodes, RBAC, and so on.
The Kubernetes LCM components handle the following custom resources (CRs):
LCMMachine
LCMCluster
HelmBundle
LCMMachine
Describes a machine that is located on a cluster.
It contains the machine type, control or worker,
StateItems that correspond to Ansible playbooks and miscellaneous actions,
for example, downloading a file or executing a shell command.
LCMMachine reflects the current state of the machine, for example,
a node IP address, and each StateItem through its status.
Multiple LCMMachine CRs can correspond to a single cluster.
LCMCluster
Describes a managed cluster. In its spec,
LCMCluster contains a set of StateItems for each type of LCMMachine,
which describe the actions that must be performed to deploy the cluster.
LCMCluster is created by the provider, using machineTypes
of the Release object. The status field of LCMCluster
reflects the status of the cluster,
for example, the number of ready or requested nodes.
HelmBundle
Wrapper for Helm charts that is handled by Helm Controller.
HelmBundle tracks what Helm charts must be installed
on a managed cluster.
LCM Controller runs on the management and regional cluster and orchestrates
the LCMMachine objects according to their type and their LCMCluster
object.
Once the LCMCluster and LCMMachine objects are created, LCM Controller
starts monitoring them to modify the spec fields and update
the status fields of the LCMMachine objects when required.
The status field of LCMMachine is updated by LCM Agent
running on a node of a management, regional, or managed cluster.
Each LCMMachine has the following lifecycle states:
Uninitialized - the machine is not yet assigned to an LCMCluster.
Pending - the agent reports a node IP address and host name.
Prepare - the machine executes StateItems that correspond
to the prepare phase. This phase usually involves downloading
the necessary archives and packages.
Deploy - the machine executes StateItems that correspond
to the deploy phase that is becoming a Mirantis Kubernetes Engine (MKE)
node.
Ready - the machine is being deployed.
Upgrade - the machine is being upgraded to the new MKE version.
Reconfigure - the machine executes StateItems that correspond
to the reconfigure phase. The machine configuration is being updated
without affecting workloads running on the machine.
The templates for StateItems are stored in the machineTypes
field of an LCMCluster object, with separate lists
for the MKE manager and worker nodes.
Each StateItem has the execution phase field for a management,
regional, and managed cluster:
The prepare phase is executed for all machines for which
it was not executed yet. This phase comprises downloading the files
necessary for the cluster deployment, installing the required packages,
and so on.
During the deploy phase, a node is added to the cluster.
LCM Controller applies the deploy phase to the nodes
in the following order:
First manager node is deployed.
The remaining manager nodes are deployed one by one
and the worker nodes are deployed in batches (by default,
up to 50 worker nodes at the same time).
LCM Controller deploys and upgrades a Mirantis Container Cloud cluster
by setting StateItems of LCMMachine objects following the corresponding
StateItems phases described above. The Container Cloud cluster upgrade
process follows the same logic that is used for a new deployment,
that is applying a new set of StateItems to the LCMMachines after
updating the LCMCluster object. But if the existing worker node is being
upgraded, LCM Controller performs draining and cordoning on this node honoring
the Pod Disruption Budgets.
This operation prevents unexpected disruptions of the workloads.
LCM Agent handles a single machine that belongs to a management, regional, or
managed cluster.
It runs on the machine operating system but communicates with apiserver
of the regional cluster. LCM Agent is deployed as a systemd unit using
cloud-init. LCM Agent has a built-in self-upgrade mechanism.
LCM Agent monitors the spec of a particular LCMMachine object
to reconcile the machine state with the object StateItems and update
the LCMMachine status accordingly. The actions that LCM Agent performs
while handling the StateItems are as follows:
Download configuration files
Run shell commands
Run Ansible playbooks in headless mode
LCM Agent provides the IP address and hostname of the machine
for the LCMMachine status parameter.
Helm Controller is used by Mirantis Container Cloud to handle
management, regional, and managed clusters core addons such as StackLight
and the application addons such as the OpenStack components.
Helm Controller is installed as a separate Helm v3 chart by the Container
Cloud provider. Its Pods are created using Deployment.
The Helm release information is stored in the KaaSRelease object for
the management and regional clusters and in the ClusterRelease object
for all types of the Container Cloud clusters.
These objects are used by the Container Cloud provider.
The Container Cloud provider uses the information from the
ClusterRelease object together with the Container Cloud API
Clusterspec. In Clusterspec, the operator can specify
the Helm release name and charts to use.
By combining the information from the ClusterproviderSpec parameter
and its ClusterRelease object, the cluster actuator generates
the LCMCluster objects. These objects are further handled by LCM Controller
and the HelmBundle object handled by Helm Controller.
HelmBundle must have the same name as the LCMCluster object
for the cluster that HelmBundle applies to.
Although a cluster actuator can only create a single HelmBundle
per cluster, Helm Controller can handle multiple HelmBundle objects
per cluster.
Helm Controller handles the HelmBundle objects and reconciles them with the
state of Helm in its cluster.
Helm Controller can also be used by the management cluster with corresponding
HelmBundle objects created as part of the initial management cluster setup.
Identity and access management (IAM) provides a central point
of users and permissions management of the Mirantis Container
Cloud cluster resources in a granular and unified manner.
Also, IAM provides infrastructure for single sign-on user experience
across all Container Cloud web portals.
IAM for Container Cloud consists of the following components:
Keycloak
Provides the OpenID Connect endpoint
Integrates with an external identity provider (IdP), for example,
existing LDAP or Google Open Authorization (OAuth)
Stores roles mapping for users
IAM Controller
Provides IAM API with data about Container Cloud projects
Handles all role-based access control (RBAC) components in Kubernetes API
IAM API
Provides an abstraction API for creating user scopes and roles
To be consistent and keep the integrity of a user database
and user permissions, in Mirantis Container Cloud,
IAM stores the user identity information internally.
However in real deployments, the identity provider usually already exists.
Out of the box, in Container Cloud, IAM supports
integration with LDAP and Google Open Authorization (OAuth).
If LDAP is configured as an external identity provider,
IAM performs one-way synchronization by mapping attributes according
to configuration.
In the case of the Google Open Authorization (OAuth) integration,
the user is automatically registered and their credentials are stored
in the internal database according to the user template configuration.
The Google OAuth registration workflow is as follows:
The user requests a Container Cloud web UI resource.
The user is redirected to the IAM login page and logs in using
the Log in with Google account option.
IAM creates a new user with the default access rights that are defined
in the user template configuration.
The user can access the Container Cloud web UI resource.
The following diagram illustrates the external IdP integration to IAM:
You can configure simultaneous integration with both external IdPs
with the user identity matching feature enabled.
Mirantis IAM performs as an OpenID Connect (OIDC) provider,
it issues a token and exposes discovery endpoints.
The credentials can be handled by IAM itself or delegated
to an external identity provider (IdP).
The issued JSON Web Token (JWT) is sufficient to perform operations across
Mirantis Container Cloud according to the scope and role defined
in it. Mirantis recommends using asymmetric cryptography for token signing
(RS256) to minimize the dependency between IAM and managed components.
When Container Cloud calls Mirantis Kubernetes Engine (MKE),
the user in Keycloak is created automatically with a JWT issued by Keycloak
on behalf of the end user.
MKE, in its turn, verifies whether the JWT is issued by Keycloak. If
the user retrieved from the token does not exist in the MKE database,
the user is automatically created in the MKE database based on the
information from the token.
The authorization implementation is out of the scope of IAM in Container
Cloud. This functionality is delegated to the component level.
IAM interacts with a Container Cloud component using the OIDC token
content that is processed by a component itself and required authorization
is enforced. Such an approach enables you to have any underlying authorization
that is not dependent on IAM and still to provide a unified user experience
across all Container Cloud components.
The following diagram illustrates the Kubernetes CLI authentication flow.
The authentication flow for Helm and other Kubernetes-oriented CLI utilities
is identical to the Kubernetes CLI flow,
but JSON Web Tokens (JWT) must be pre-provisioned.
Mirantis Container Cloud uses StackLight, the logging,
monitoring, and alerting solution that provides a single pane of glass
for cloud maintenance and day-to-day operations as well as
offers critical insights into cloud health including operational information
about the components deployed in management, regional, and managed clusters.
StackLight is based on Prometheus, an open-source monitoring solution and a
time series database.
Mirantis Container Cloud deploys the StackLight stack
as a release of a Helm chart that contains the helm-controller
and helmbundles.lcm.mirantis.com (HelmBundle) custom resources.
The StackLight HelmBundle consists of a set of Helm charts
with the StackLight components that include:
Receives, consolidates, and deduplicates the alerts sent by Alertmanager
and visually represents them through a simple web UI. Using the Alerta
web UI, you can view the most recent or watched alerts, group, and
filter alerts.
Alertmanager
Handles the alerts sent by client applications such as Prometheus,
deduplicates, groups, and routes alerts to receiver integrations.
Using the Alertmanager web UI, you can view the most recent fired
alerts, silence them, or view the Alertmanager configuration.
Elasticsearch Curator
Maintains the data (indexes) in OpenSearch by performing
such operations as creating, closing, or opening an index as well as
deleting a snapshot. Also, manages the data retention policy in
OpenSearch.
Elasticsearch Exporter Compatible with OpenSearch
The Prometheus exporter that gathers internal OpenSearch metrics.
Grafana
Builds and visually represents metric graphs based on time series
databases. Grafana supports querying of Prometheus using the PromQL
language.
Database back ends
StackLight uses PostgreSQL for Alerta and Grafana. PostgreSQL reduces
the data storage fragmentation while enabling high availability.
High availability is achieved using Patroni, the PostgreSQL cluster
manager that monitors for node failures and manages failover
of the primary node. StackLight also uses Patroni to manage major
version upgrades of PostgreSQL clusters, which allows leveraging
the database engine functionality and improvements
as they are introduced upstream in new releases,
maintaining functional continuity without version lock-in.
Logging stack
Responsible for collecting, processing, and persisting logs and
Kubernetes events. By default, when deploying through the Container
Cloud web UI, only the metrics stack is enabled on managed clusters. To
enable StackLight to gather managed cluster logs, enable the logging
stack during deployment. On management clusters, the logging stack is
enabled by default. The logging stack components include:
OpenSearch, which stores logs and notifications.
Fluentd-logs, which collects logs, sends them to OpenSearch, generates
metrics based on analysis of incoming log entries, and exposes these
metrics to Prometheus.
OpenSearch Dashboards, which provides real-time visualization of
the data stored in OpenSearch and enables you to detect issues.
Metricbeat, which collects Kubernetes events and sends them to
OpenSearch for storage.
Prometheus-es-exporter, which presents the OpenSearch data
as Prometheus metrics by periodically sending configured queries to
the OpenSearch cluster and exposing the results to a scrapable HTTP
endpoint like other Prometheus targets.
Optional. Cerebro, a web UI for managing the OpenSearch cluster. Using
the Cerebro web UI, you can get a detailed view on your OpenSearch
cluster and debug issues. Cerebro is disabled
by default.
Note
The logging mechanism performance depends on the cluster log load. In
case of a high load, you may need to increase the default resource requests
and limits for fluentdLogs. For details, see
StackLight configuration parameters: Resource limits.
Metric collector
Collects telemetry data (CPU or memory usage, number of active alerts,
and so on) from Prometheus and sends the data to centralized cloud
storage for further processing and analysis. Metric collector runs on
the management cluster.
Note
This component is designated for internal StackLight use only.
Prometheus
Gathers metrics. Automatically discovers and monitors the endpoints.
Using the Prometheus web UI, you can view simple visualizations and
debug. By default, the Prometheus database stores metrics of the past 15
days or up to 15 GB of data depending on the limit that is reached
first.
Prometheus Blackbox Exporter
Allows monitoring endpoints over HTTP, HTTPS, DNS, TCP, and ICMP.
Prometheus-es-exporter
Presents the OpenSearch data as Prometheus metrics by periodically
sending configured queries to the OpenSearch cluster and exposing the
results to a scrapable HTTP endpoint like other Prometheus targets.
Prometheus Node Exporter
Gathers hardware and operating system metrics exposed by kernel.
Prometheus Relay
Adds a proxy layer to Prometheus to merge the results from underlay
Prometheus servers to prevent gaps in case some data is missing on
some servers. Is available only in the HA StackLight mode.
Reference Application Available since 2.21.0
Enables workload monitoring on non-MOSK managed clusters.
Mimics a classical microservice application and provides metrics that
describe the likely behavior of user workloads.
Enables sending Alertmanager notifications to Salesforce to allow
creating Salesforce cases and closing them once the alerts are resolved.
Disabled by default.
Salesforce reporter
Queries Prometheus for the data about the amount of vCPU, vRAM, and
vStorage used and available, combines the data, and sends it to
Salesforce daily. Mirantis uses the collected data for further analysis
and reports to improve the quality of customer support. Disabled by
default.
Telegraf
Collects metrics from the system. Telegraf is plugin-driven and has
the concept of two distinct set of plugins: input plugins collect
metrics from the system, services, or third-party APIs; output plugins
write and expose metrics to various destinations.
The Telegraf agents used in Container Cloud include:
telegraf-ds-smart monitors SMART disks, and runs on both
management and managed clusters.
telegraf-ironic monitors Ironic on the baremetal-based
management clusters. The ironic input plugin collects and
processes data from Ironic HTTP API, while the http_response
input plugin checks Ironic HTTP API availability. As an output plugin,
to expose collected data as Prometheus target, Telegraf uses
prometheus.
telegraf-docker-swarm gathers metrics from the Mirantis Container
Runtime API about the Docker nodes, networks, and Swarm services. This
is a Docker Telegraf input plugin with downstream additions.
Telemeter
Enables a multi-cluster view through a Grafana dashboard of the
management cluster. Telemeter includes a Prometheus federation push
server and clients to enable isolated Prometheus instances, which
cannot be scraped from a central Prometheus instance, to push metrics
to the central location.
The Telemeter services are distributed as follows:
Management cluster hosts the Telemeter server
Regional clusters host the Telemeter server and Telemeter client
Managed clusters host the Telemeter client
The metrics from managed clusters are aggregated on regional clusters.
Then both regional and managed clusters metrics are sent from regional
clusters to the management cluster.
Note
This component is designated for internal StackLight use only.
Every Helm chart contains a default values.yml file. These default values
are partially overridden by custom values defined in the StackLight Helm chart.
Before deploying a managed cluster, you can select the HA or non-HA StackLight
architecture type. The non-HA mode is set by default. On the management and
regional clusters, StackLight is deployed in the HA mode only.
The following table lists the differences between the HA and non-HA modes:
One persistent volume is provided for storing data. In case of a service
or node failure, a new pod is redeployed and the volume is reattached to
provide the existing data. Such setup has a reduced hardware footprint
but provides less performance.
Two Prometheus instances
Three OpenSearch instances
Three PostgreSQL instances
Two iam-proxy instances
Since 2.23.0 and 2.23.1 for MOSK 23.1
Local Volume Provisioner is used to provide local host storage. In case
of a service or node failure, the traffic is automatically redirected to
any other running Prometheus or OpenSearch server. For better
performance, Mirantis recommends that you deploy StackLight in the HA
mode. Two iam-proxy instances ensure access to HA components if one
iam-proxy node fails.
Depending on the Container Cloud cluster type and selected StackLight database
mode, StackLight is deployed on the following number of nodes:
StackLight provides five web UIs including Prometheus, Alertmanager, Alerta,
OpenSearch Dashboards, and Grafana. Access to StackLight web UIs is protected
by Keycloak-based Identity and access management (IAM). All web UIs except
Alerta are exposed to IAM through the IAM proxy middleware. The Alerta
configuration provides direct integration with IAM.
The following diagram illustrates accessing the IAM-proxied StackLight web UIs,
for example, Prometheus web UI:
Authentication flow for the IAM-proxied StackLight web UIs:
A user enters the public IP of a StackLight web UI, for example, Prometheus
web UI.
The public IP leads to IAM proxy, deployed as a Kubernetes LoadBalancer,
which protects the Prometheus web UI.
LoadBalancer routes the HTTP request to Kubernetes internal IAM proxy
service endpoints, specified in the X-Forwarded-Proto or X-Forwarded-Host
headers.
The Keycloak login form opens (the login_url field in the IAM proxy
configuration, which points to Keycloak realm) and the user enters
the user name and password.
Keycloak validates the user name and password.
The user obtains access to the Prometheus web UI (the upstreams field
in the IAM proxy configuration).
Note
The discovery URL is the URL of the IAM service.
The upstream URL is the hidden endpoint of a web UI (Prometheus web UI in
the example above).
The following diagram illustrates accessing the Alerta web UI:
Authentication flow for the Alerta web UI:
A user enters the public IP of the Alerta web UI.
The public IP leads to Alerta deployed as a Kubernetes LoadBalancer type.
LoadBalancer routes the HTTP request to the Kubernetes internal Alerta
service endpoint.
The Keycloak login form opens (Alerta refers to the IAM realm) and
the user enters the user name and password.
Using the Mirantis Container Cloud web UI,
on the pre-deployment stage of a managed cluster,
you can view, enable or disable, or tune the following StackLight features
available:
StackLight HA mode.
Database retention size and time for Prometheus.
Tunable index retention period for OpenSearch.
Tunable PersistentVolumeClaim (PVC) size for Prometheus and OpenSearch
set to 16 GB for Prometheus and 30 GB for OpenSearch by
default. The PVC size must be logically aligned with the retention periods or
sizes for these components.
Email and Slack receivers for the Alertmanager notifications.
Predefined set of dashboards.
Predefined set of alerts and capability to add
new custom alerts for Prometheus in the following exemplary format:
StackLight measures, analyzes, and reports in a timely manner about failures
that may occur in the following Mirantis Container Cloud
components and their sub-components, if any:
The data collected and transmitted through an encrypted channel back to
Mirantis provides our Customer Success Organization information to better
understand the operational usage patterns our customers are experiencing
as well as to provide feedback on product usage statistics to enable our
product teams to enhance our products and services for our customers.
Since Container Cloud 2.21.0, the summary of all deployed Container Cloud
configurations is also collected. The data is anonymized from all sensitive
information, such as IDs, IP addresses, passwords, private keys, and so on.
Mirantis collects information about the following Container Cloud objects,
if any:
Cluster
Machine
MachinePool
MCCUpgrade
BareMetalHost
BareMetalHostProfile
IPAMHost
IPAddr
KaaSCephCluster
L2Template
Subnet
The node-level resource data are broken down into three broad categories:
Cluster, Node, and Namespace. The telemetry data tracks Allocatable,
Capacity, Limits, Requests, and actual Usage of node-level resources.
StackLight components, which require external access, automatically use the
same proxy that is configured for Mirantis Container Cloud clusters. Therefore,
you only need to configure proxy during deployment of your management,
regional, or managed clusters. No additional actions are required to set up
proxy for StackLight. For more details about implementation of proxy support in
Container Cloud, see Proxy and cache support.
Note
Proxy handles only the HTTP and HTTPS traffic. Therefore, for
clusters with limited or no Internet access, it is not possible to set up
Alertmanager email notifications, which use SMTP, when proxy is used.
Proxy is used for the following StackLight components:
Component
Cluster type
Usage
Alertmanager
Any
As a default http_config
for all HTTP-based receivers except the predefined HTTP-alerta and
HTTP-salesforce. For these receivers, http_config is overridden on
the receiver level.
Metric Collector
Management
To send outbound cluster metrics to Mirantis.
Salesforce notifier
Any
To send notifications to the Salesforce instance.
Salesforce reporter
Any
To send metric reports to the Salesforce instance.
Telemeter client
Regional
To send all metrics from the clusters of a region, including the managed
and regional clusters, to the management cluster. Proxy is not used for
the Telemeter client on managed clusters because managed clusters must
have a direct access to their regional cluster.
Reference Application is a small microservice application that enables
workload monitoring on non-MOSK managed clusters. It mimics a
classical microservice application and provides metrics that describe the
likely behavior of user workloads.
The application consists of the following API and database services that allow
putting simple records into the database through the API and retrieving them:
Reference Application API
Runs on StackLight nodes and provides API access to the database.
Runs three API instances for high availability.
PostgreSQL Since Container Cloud 2.22.0
Runs on worker nodes and stores the data on an attached PersistentVolumeClaim
(PVC). Runs three database instances for high availability.
Note
Before version 2.22.0, Container Cloud used MariaDB as the
database management system instead of PostgreSQL.
StackLight queries the API measuring response times for each query.
No caching is being done, so each API request must go to the database,
allowing to verify the availability of a stateful workload on the cluster.
Reference Application requires the following resources on top of the main
product requirements:
Using Mirantis Container Cloud, you can deploy a Mirantis Kubernetes Engine
(MKE) cluster on bare metal, OpenStack, or VMware vSphere cloud providers.
Each cloud provider requires corresponding resources.
Note
Using the free Mirantis license, you can create up to three
Container Cloud managed clusters
with three worker nodes on each cluster.
Within the same quota, you can also attach existing
MKE clusters that are not deployed by Container Cloud.
If you need to increase this quota,
contact Mirantis support for further details.
A bootstrap node is necessary only to deploy the management cluster.
When the bootstrap is complete, the bootstrap node can be
redeployed and its resources can be reused
for the managed cluster workloads.
The minimum reference system requirements of a baremetal-based bootstrap
seed node are described in System requirements for the seed node.
The minimum reference system requirements a bootstrap node for other supported
Container Cloud providers are as follows:
Any local machine on Ubuntu 20.04 that requires access to the provider API
with the following configuration:
2 vCPUs
4 GB of RAM
5 GB of available storage
Docker version currently available for Ubuntu 20.04
Internet access for downloading of all required artifacts
Note
For the vSphere cloud provider, you can use RHEL 7.9 as the
operating system for the bootstrap node. The system requirements
are the same as for Ubuntu.
If you use a firewall or proxy, make sure that the bootstrap, management,
and regional clusters have access to the following IP ranges and domain names
required for the Container Cloud content delivery network and alerting:
The following hardware configuration is used as a reference to deploy
Mirantis Container Cloud with bare metal Container Cloud clusters with
Mirantis Kubernetes Engine.
Reference hardware configuration for Container Cloud
management and managed clusters on bare metal¶
Three manager nodes for HA and three workerstorage
nodes for a minimal Ceph cluster. For more details about Ceph
requirements, see Management cluster storage.
A management cluster requires 2 volumes for Container Cloud
(total 50 GB) and 5 volumes for StackLight (total 60 GB).
A managed cluster requires 5 volumes for StackLight.
The seed node is necessary only to deploy the management cluster.
When the bootstrap is complete, the bootstrap node can be
redeployed and its resources can be reused
for the managed cluster workloads.
The minimum reference system requirements for a baremetal-based bootstrap
seed node are as follows:
Basic server on Ubuntu 20.04 with the following configuration:
Kernel version 4.15.0-76.86 or later
8 GB of RAM
4 CPU
10 GB of free disk space for the bootstrap cluster cache
No DHCP or TFTP servers on any NIC networks
Routable access IPMI network for the hardware servers. For more details, see
Host networking.
Internet access for downloading of all required artifacts
The following diagram illustrates the physical and virtual L2 underlay
networking schema for the final state of the Mirantis Container Cloud
bare metal deployment.
The network fabric reference configuration is a spine/leaf with 2 leaf ToR
switches and one out-of-band (OOB) switch per rack.
Reference configuration uses the following switches for ToR and OOB:
Cisco WS-C3560E-24TD has 24 of 1 GbE ports. Used in OOB network
segment.
Dell Force 10 S4810P has 48 of 1/10GbE ports. Used as ToR in Common/PXE
network segment.
In the reference configuration, all odd interfaces from NIC0 are connected
to TORSwitch1, and all even interfaces from NIC0 are connected
to TORSwitch2. The Baseboard Management Controller (BMC) interfaces
of the servers are connected to OOBSwitch1.
The following recommendations apply to all types of nodes:
Use the Link Aggregation Control Protocol (LACP) bonding mode
with MC-LAG domains configured on leaf switches. This corresponds to
the 802.3ad bond mode on hosts.
Use ports from different multi-port NICs when creating bonds. This makes
network connections redundant if failure of a single NIC occurs.
Configure the ports that connect servers to the PXE network with PXE VLAN
as native or untagged. On these ports, configure LACP fallback to ensure
that the servers can reach DHCP server and boot over network.
The management cluster requires minimum two storage devices per node.
Each device is used for different type of storage.
The first device is always used for boot partitions and the root
file system. SSD is recommended. RAID device is not supported.
One storage device per server is reserved for local persistent
volumes. These volumes are served by the Local Storage Static Provisioner
(local-volume-provisioner) and used by many services of Container Cloud.
While planning the deployment of an OpenStack-based Mirantis Container Cloud
cluster with Mirantis Kubernetes Engine (MKE), consider the following general
requirements:
Kubernetes on OpenStack requires the Cinder API V3 and Octavia API
availability.
Mirantis supports deployments based on OpenStack Victoria or Yoga with
Open vSwitch (OVS) or Tungsten Fabric (TF) on top of Mirantis OpenStack for Kubernetes
(MOSK) Victoria or Yoga with TF.
If you use a firewall or proxy, make sure that the bootstrap, management,
and regional clusters have access to the following IP ranges and domain names
required for the Container Cloud content delivery network and alerting:
mirror.mirantis.com and repos.mirantis.com for packages
binary.mirantis.com for binaries and Helm charts
mirantis.azurecr.io and *.blob.core.windows.net for Docker images
mcc-metrics-prod-ns.servicebus.windows.net:9093 for Telemetry
(port 443 if proxy is enabled)
mirantis.my.salesforce.com and login.salesforce.com
for Salesforce alerts
Note
Access to Salesforce is required from any Container Cloud
cluster type.
If any additional Alertmanager notification receiver is enabled,
for example, Slack, its endpoint must also be accessible
from the cluster.
Note
The requirements in this section apply to the latest supported
Container Cloud release.
Requirements for an OpenStack-based Container Cloud cluster¶
Resource
Management or regional cluster
Managed cluster
Comments
# of nodes
3 (HA) + 1 (Bastion)
5 (6 with StackLight HA)
A bootstrap cluster requires access to the OpenStack API.
Each management or regional cluster requires 3 nodes for the manager nodes HA.
Adding more than 3 nodes to a management or regional cluster is not supported.
A managed cluster requires 3 manager nodes for HA and 2 worker nodes for the
Container Cloud workloads. If the multiserver mode is enabled for StackLight,
3 worker nodes are required for workloads.
Each management or regional cluster requires 1 node for the Bastion instance
that is created with a public IP address to allow SSH access to instances.
# of vCPUs per node
8
8
The Bastion node requires 1 vCPU.
Refer to the RAM recommendations described below to plan resources
for different types of nodes.
RAM in GB per node
24
16
To prevent issues with low RAM, Mirantis recommends the following types
of instances for a managed cluster with 50-200 nodes:
16 vCPUs and 32 GB of RAM - manager node
16 vCPUs and 128 GB of RAM - nodes where the StackLight server components run
The Bastion node requires 1 GB of RAM.
Storage in GB per node
120
120
For the Bastion node, the default amount of storage is enough
To boot machines from a block storage volume, verify that disks
performance matches the etcd requirements as described in
etcd documentation
To boot the Bastion node from a block storage volume, 80 GB is enough
If you use a firewall or proxy, make sure that the bootstrap, management,
and regional clusters have access to the following IP ranges and domain names
required for the Container Cloud content delivery network and alerting:
mirror.mirantis.com and repos.mirantis.com for packages
binary.mirantis.com for binaries and Helm charts
mirantis.azurecr.io and *.blob.core.windows.net for Docker images
mcc-metrics-prod-ns.servicebus.windows.net:9093 for Telemetry
(port 443 if proxy is enabled)
mirantis.my.salesforce.com and login.salesforce.com
for Salesforce alerts
Note
Access to Salesforce is required from any Container Cloud
cluster type.
If any additional Alertmanager notification receiver is enabled,
for example, Slack, its endpoint must also be accessible
from the cluster.
Note
The requirements in this section apply to the latest supported
Container Cloud release.
Requirements for a vSphere-based Container Cloud cluster¶
Resource
Management cluster
Managed cluster
Comments
# of nodes
3 (HA)
5 (6 with StackLight HA)
A bootstrap cluster requires access to the vSphere API.
A management cluster requires 3 nodes for the manager nodes HA. Adding
more than 3 nodes to a management or regional cluster is not supported.
A managed cluster requires 3 manager nodes for HA and 2 worker nodes for the
Container Cloud workloads. If the multiserver mode is enabled for StackLight,
3 worker nodes are required for workloads.
# of vCPUs per node
8
8
Refer to the RAM recommendations described below to plan resources
for different types of nodes.
RAM in GB per node
24
16
To prevent issues with low RAM, Mirantis recommends the following VM
templates for a managed cluster with 50-200 nodes:
16 vCPUs and 32 GB of RAM - manager node
16 vCPUs and 128 GB of RAM - nodes where the StackLight server components run
Storage in GB per node
120
120
The listed amount of disk space must be available as a shared
datastore of any type, for example, NFS or vSAN, mounted on all
hosts of the vCenter cluster.
For a management and managed cluster, a base OS VM template
must be present in the VMware VM templates folder
available to Container Cloud. For details about the template,
see Prepare the virtual machine template.
This license type allows running unlimited guests inside one hypervisor.
The amount of licenses is equal to the amount of hypervisors in
vCenter Server, which will be used to host RHEL-based machines.
Container Cloud will schedule machines according to scheduling rules
applied to vCenter Server. Therefore, make sure that your
RedHat Customer portal account has enough licenses for allowed
hypervisors.
MCR
20.10.13
20.10.13
Mirantis Container Runtime (MCR) is deployed by Container Cloud as a
Container Runtime Interface (CRI) instead of Docker Engine.
A shared datastore must be mounted on all hosts of the
vCenter cluster. Combined with Distributed Resources Scheduler (DRS),
it ensures that the VMs are dynamically scheduled to the cluster hosts.
RHEL 7.8 deployment is possible with allowed access to the
rhel-7-server-rpms repository provided by the Red Hat Enterprise
Linux Server 7 x86_64.
Verify that your RHEL license or activation key meets this requirement.
CentOS 7.9 and RHEL 8.4 deployments are available as Technology
Preview. Use this configuration for testing and evaluation purposes
only.
A Container Cloud cluster based on mixed operating systems, such as
RHEL and CentOS, or on mixed versions of RHEL, such as RHEL 7.9 and
8.4, is not supported.
While planning the attachment of an existing Mirantis Kubernetes Engine (MKE)
cluster that is not deployed by Container Cloud, consider the following
general requirements for StackLight:
For StackLight in non-HA mode, make sure that you have the default storage
class configured on the MKE cluster being attached. To select and configure
a persistent storage for StackLight, refer to MKE documentation: Persistent
Kubernetes storage.
While planning the attachment of an existing Mirantis Kubernetes Engine (MKE)
cluster that is not deployed by Container Cloud, consider the following cluster
size requirements for StackLight. Depending on the following specific
StackLight HA and logging settings, use the example size guidelines below:
The non-HA mode - StackLight services are installed on a minimum of one node
with the StackLight label (StackLight nodes) with no redundancy
using Persistent Volumes (PVs) from the default storage class to store data.
Metric collection agents are installed on each node (Other nodes).
The HA mode - StackLight services are installed on a minimum of three nodes
with the StackLight label (StackLight nodes) with redundancy
using PVs provided by Local Volume Provisioner to store data. Metric
collection agents are installed on each node (Other nodes).
Logging enabled - the Enable logging option is turned on, which
enables the OpenSearch cluster to store infrastructure logs.
Logging disabled - the Enable logging option is turned off. In
this case, StackLight will not install OpenSearch and will not collect
infrastructure logs.
LoadBalancer (LB) Services support is required to provide external access
to StackLight web UIs.
StackLight requirements for an attached MKE cluster, with logging enabled:¶
In the non-HA mode, StackLight components are bound to the nodes labeled
with the StackLight label. If there are no nodes labeled, StackLight
components will be scheduled to all schedulable worker nodes until the
StackLight label(s) are added. The requirements presented in the
table for the non-HA mode are summarized requirements for all StackLight
nodes.
If you require all Internet access to go through a proxy server
for security and audit purposes, you can bootstrap management and regional
clusters using proxy. The proxy server settings consist of three standard
environment variables that are set prior to the bootstrap process:
HTTP_PROXY
HTTPS_PROXY
NO_PROXY
These settings are not propagated to managed clusters. However, you can enable
a separate proxy access on a managed cluster using the Container Cloud web UI.
This proxy is intended for the end user needs and is not used for a managed
cluster deployment or for access to the Mirantis resources.
Caution
Since Container Cloud uses the OpenID Connect (OIDC) protocol
for IAM authentication, management clusters require
a direct non-proxy access from regional and managed clusters.
StackLight components, which require external access, automatically use the
same proxy that is configured for Container Cloud clusters.
On the managed clusters with limited Internet access, a proxy is required for
StackLight components that use HTTP and HTTPS and are disabled by default but
need external access if enabled, for example, for the Salesforce integration
and Alertmanager notifications external rules.
For more details about proxy implementation in StackLight, see StackLight proxy.
For the list of Mirantis resources and IP addresses to be accessible
from the Container Cloud clusters, see Hardware and system requirements.
After enabling proxy support on regional and managed clusters, proxy is used
for:
The Container Cloud managed clusters are deployed without direct Internet
access in order to consume less Internet traffic in your cloud.
The Mirantis artifacts used during managed clusters deployment are downloaded
through a cache running on a regional cluster.
The feature is enabled by default on new managed clusters
and will be automatically enabled on existing clusters during upgrade
to the latest version.
Caution
IAM operations require a direct non-proxy access
of a managed cluster to a management cluster.
Ceph monitors use their node host networks to interact with Ceph daemons.
Ceph daemons communicate with each other over a specified cluster network
and provide endpoints over the public network.
The messenger V2 (msgr2) or earlier V1 (msgr) protocols are used for
communication between Ceph daemons.
Ceph daemon
Network
Protocol
Port
Description
Consumers
Manager (mgr)
Cluster network
msgr/msgr2
6800
Listens on the first available port of the 6800-7300 range
csi-rbdplugin,
csi-rbdprovisioner,
rook-ceph-mon
Metadata server (mds)
Cluster network
msgr/msgr2
6800
Listens on the first available port of the 6800-7300 range
csi-cephfsplugin,
csi-cephfsprovisioner
Monitor (mon)
LCM host network
msgr/msgr2
msgr:3300,
msgr2:6789
Monitor has separate ports for msgr and msgr2
Ceph clients
rook-ceph-osd,
rook-ceph-rgw
Ceph OSD (osd)
Cluster network
msgr/msgr2
6800-7300
Binds to the first available port from the 6800-7300 range
To ensure the Mirantis Container Cloud stability in managing
the Container Cloud-based Mirantis Kubernetes Engine (MKE) clusters,
the following MKE API functionality is not available for the
Container Cloud-based MKE clusters as compared to the attached MKE clusters
that are not deployed by Container Cloud.
Use the Container Cloud web UI or CLI for this functionality instead.
Public APIs limitations in a Container Cloud-based MKE cluster¶
API endpoint
Limitation
GET/swarm
Swarm Join Tokens are filtered out for all users, including admins.
PUT/api/ucp/config-toml
All requests are forbidden.
POST/nodes/{id}/update
Requests for the following changes are forbidden:
Change Role
Add or remove the com.docker.ucp.orchestrator.swarm and
com.docker.ucp.orchestrator.kubernetes labels.
The bare metal management system enables the Infrastructure Operator
to deploy Mirantis Container Cloud on a set of bare metal
servers.
It also enables Container Cloud to deploy managed clusters on bare metal
servers without a pre-provisioned operating system.
The Infrastructure Operator performs the following steps to install
Container Cloud in a bare metal environment:
The baremetal-based Container Cloud does not manage
the underlay networking fabric but requires
specific network configuration to operate.
Install Ubuntu 20.04 on one of the bare metal machines to create a seed
node and copy the bootstrap tarball to this node.
Obtain the Mirantis license file that will be required during the bootstrap.
Create the deployment configuration files that include the bare metal hosts
metadata.
Validate the deployment templates using fast preflight.
Run the bootstrap script for the fully automated installation of the
management cluster onto the selected bare metal hosts.
Using the bootstrap script, the Container Cloud bare metal management
system prepares the seed node for the management cluster
and starts the deployment of Container Cloud itself. The bootstrap script
performs all necessary operations to perform the automated management cluster
setup. The deployment diagram below illustrates the bootstrap workflow
of a baremetal-based management cluster.
Install basic Ubuntu 20.04 server using standard installation images of the
operating system on the bare metal seed node.
Log in to the seed node that is running Ubuntu 20.04.
Prepare the system and network configuration:
Create a virtual bridge to connect to your PXE network on the
seed node. Use the following netplan-based configuration file
as an example:
# cat /etc/netplan/config.yamlnetwork:version:2renderer:networkdethernets:ens3:dhcp4:falsedhcp6:falsebridges:br0:addresses:# Please, adjust for your environment-10.0.0.15/24dhcp4:falsedhcp6:false# Please, adjust for your environmentgateway4:10.0.0.1interfaces:# Interface name may be different in your environment-ens3nameservers:addresses:# Please, adjust for your environment-8.8.8.8parameters:forward-delay:4stp:false
Apply the new network configuration using netplan:
The system output must contain a json file with no error messages.
In case of errors, follow the steps provided in Troubleshooting.
Note
If you require all Internet access to go through a proxy server
for security and audit purposes, configure Docker proxy settings
as described in the official
Docker documentation.
Verify that the seed node has direct access to the Baseboard Management
Controller (BMC) of each bare metal host. All target hardware nodes must
be in the poweroff state.
Before adding new BareMetalHost objects, configure hardware hosts to
correctly load them over the PXE network.
Important
Consider the following common requirements for hardware hosts
configuration:
Update firmware for BIOS and Baseboard Management Controller (BMC) to the
latest available version, especially if you are going to apply the UEFI
configuration.
Container Cloud uses the ipxe.efi binary loader that might be not
compatible with old firmware and have vendor-related issues with UEFI
booting. For example, the Supermicro issue.
In this case, we recommend using the legacy booting format.
Configure all or at least the PXE NIC on switches.
If the hardware host has more than one PXE NIC to boot, we strongly
recommend setting up only one in the boot order. It speeds up the
provisioning phase significantly.
Some hardware vendors require a host to be rebooted during BIOS
configuration changes from legacy to UEFI or vice versa for the
extra option with NIC settings to appear in the menu.
Connect only one Ethernet port on a host to the PXE network at any given
time. Collect the physical address (MAC) of this interface and use it to
configure the BareMetalHost object describing the host.
To configure BIOS on a bare metal host:
Legacy hardware host configuration
Enable the global BIOS mode using
BIOS > Boot > boot mode select > legacy. Reboot the host
if required.
Enable the LAN-PXE-OPROM support using the following menus:
UEFI support might not apply to all NICs. But at least built-in
network interfaces should support it.
Set up the configured boot order:
BIOS > Boot > UEFI-Boot-Order#1 > UEFI Hard Disk
BIOS > Boot > UEFI-Boot-Order#1 > UEFI Network
Save changes and power off the host.
Prepare metadata and deploy the management cluster¶
Using the example procedure below, replace the addresses and credentials
in the configuration YAML files with the data from your environment.
Keep everything else as is, including the file names and YAML structure.
The overall network mapping scheme with all L2/L3 parameters, for example,
for a single 10.0.0.0/24 network, is described in the following table.
The configuration of each parameter indicated in this table is described
in the steps below.
Log in to your account and download the mirantis.lic license file.
Save the license file as mirantis.lic under the kaas-bootstrap
directory on the bootstrap node.
Verify that mirantis.lic contains the exact Container Cloud license
previously downloaded from www.mirantis.com
by decoding the license JWT token, for example, using jwt.io.
Example of a valid decoded Container Cloud license data with the mandatory
license field:
Update the cluster definition template in
templates/bm/cluster.yaml.template
according to the environment configuration. Use the table below.
Manually set all parameters that start with SET_. For example,
SET_METALLB_ADDR_POOL.
The IP address of the externally accessible API endpoint
of the cluster. This address must NOT be
within the SET_METALLB_ADDR_POOL range but must be within
the PXE/Management network. External load balancers are not supported.
10.0.0.90
SET_METALLB_ADDR_POOL
The IP range to be used as external load balancers for the Kubernetes
services with the LoadBalancer type. This range must be within
the PXE/Management network. The minimum required range is 19 IP addresses.
10.0.0.61-10.0.0.80
Configure NTP server.
Before Container Cloud 2.23.0, optional if servers from the Ubuntu NTP pool
(*.ubuntu.pool.ntp.org) are accessible from the node where
your cluster is being provisioned. Otherwise, configure the
regional NTP server parameters as described below.
Since Container Cloud 2.23.0, optionally disable NTP that is enabled by default. This option disables the
management of chrony configuration by Container Cloud to use your own
system for chrony management. Otherwise, configure the regional NTP server
parameters as described below.
NTP configuration
Configure the regional NTP server parameters to be applied to all machines
of regional and managed clusters in the specified region.
In templates/bm/cluster.yaml.template, add the ntp:servers section
with the list of required server names:
Inspect the default bare metal host profile definition in
templates/bm/baremetalhostprofiles.yaml.template.
If your hardware configuration differs from the reference,
adjust the default profile to match. For details, see
Customize the default bare metal host profile.
Warning
All data will be wiped during cluster deployment on devices
defined directly or indirectly in the fileSystems list of
BareMetalHostProfile. For example:
A raw device partition with a file system on it
A device partition in a volume group with a logical volume that has a
file system on it
An mdadm RAID device with a file system on it
An LVM RAID device with a file system on it
The wipe field is always considered true for these devices.
The false value is ignored.
Therefore, to prevent data loss, move the necessary data from these file
systems to another server beforehand, if required.
Update the bare metal hosts definition template in
templates/bm/baremetalhosts.yaml.template
according to the environment configuration. Use the table below.
Manually set all parameters that start with SET_.
The MAC address of the first master node in the PXE network.
ac:1f:6b:02:84:71
SET_MACHINE_0_BMC_ADDRESS
The IP address of the BMC endpoint for the first master node in
the cluster. Must be an address from the OOB network
that is accessible through the PXE network default gateway.
The MAC address of the second master node in the PXE network.
ac:1f:6b:02:84:72
SET_MACHINE_1_BMC_ADDRESS
The IP address of the BMC endpoint for the second master node in
the cluster. Must be an address from the OOB network
that is accessible through the PXE network default gateway.
The MAC address of the third master node in the PXE network.
ac:1f:6b:02:84:73
SET_MACHINE_2_BMC_ADDRESS
The IP address of the BMC endpoint for the third master node in
the cluster. Must be an address from the OOB network
that is accessible through the PXE network default gateway.
Since Container Cloud 2.21.0, a user name and password in plain
text are required.
Before Container Cloud 2.21.0, the Base64 encoding of a user name
and password is required. You can obtain the Base64-encoded user
name and password using the following command in your Linux
console:
$echo-n<username|password>|base64
Update the Subnet objects definition template in
templates/bm/ipam-objects.yaml.template
according to the environment configuration. Use the table below.
Manually set all parameters that start with SET_.
For example, SET_IPAM_POOL_RANGE.
The IP address of the externally accessible API endpoint
of the cluster. This address must NOT be
within the SET_METALLB_ADDR_POOL range but must be within the
PXE/Management network. External load balancers are not supported.
The IP address range to be used as external load balancers for the
Kubernetes services with the LoadBalancer type. This range must
be within the PXE/Management network. The minimum required range is
19 IP addresses.
Use the same value that you used for this parameter in the
cluster.yaml.template file (see above).
Optional. To configure the separated PXE and management networks instead of
one PXE/management network, proceed to Separate PXE and management networks.
Optional. To connect the cluster hosts to the PXE/Management
network using bond interfaces, proceed to Configure NIC bonding.
If you require all Internet access to go through a proxy server,
in bootstrap.env, add the following environment variables to bootstrap
the cluster using proxy:
The provisioning IP address. This address will be assigned to the
interface of the seed node defined by the KAAS_BM_PXE_BRIDGE
parameter (see below). The PXE service of the bootstrap cluster will
use this address to network boot the bare metal hosts for the
cluster.
10.0.0.20
KAAS_BM_PXE_MASK
The CIDR prefix for the PXE network. It will be used with KAAS_BM_PXE_IP
address when assigning it to network interface.
24
KAAS_BM_PXE_BRIDGE
The PXE network bridge name. The name must match the name
of the bridge created on the seed node during the
Prepare the seed node stage.
br0
KAAS_BM_BM_DHCP_RANGE
The start_ip and end_ip addresses must be within the PXE network.
This range will be used by dnsmasq to provide IP addresses for nodes
during provisioning.
10.0.0.30,10.0.0.49,255.255.255.0
BOOTSTRAP_METALLB_ADDRESS_POOL
The pool of IP addresses that will be used by services
in the bootstrap cluster. Can be the same as the
SET_METALLB_ADDR_POOL range for the cluster, or a different range.
10.0.0.61-10.0.0.80
Run the verification preflight script to validate the deployment
templates configuration:
./bootstrap.shpreflight
The command outputs a human-readable report with the verification details.
The report includes the list of verified bare metal nodes and their
ChassisPower status.
This status is based on the deployment templates configuration used
during the verification.
Caution
If the report contains information about missing dependencies
or incorrect configuration, fix the issues before proceeding
to the next step.
Optional. Enable infinite timeout for all bootstrap stages by exporting the
following environment variable or adding it to bootstrap.env:
exportKAAS_BOOTSTRAP_INFINITE_TIMEOUT=true
Infinite timeout prevents the bootstrap failure due to timeout. This option
is useful in the following cases:
The network speed is slow for artifacts downloading
An infrastructure configuration does not allow booting fast
A bare-metal node inspecting presupposes more than two HDDSATA disks
to attach to a machine
Optional. Available since Container Cloud 2.23.0. Customize the cluster and
region name by exporting the following environment variables or adding them
to bootstrap.env:
The Keycloak URL that the system outputs when the bootstrap completes.
The admin password for Keycloak is located in
kaas-bootstrap/passwords.yml along with other IAM passwords.
Note
The Container Cloud web UI and StackLight endpoints are available
through Transport Layer Security (TLS) and communicate with Keycloak
to authenticate users. Keycloak is exposed using HTTPS and
self-signed TLS certificates that are not trusted by web browsers.
You can configure L2 templates for the management cluster to set up
a bond network interface for the PXE/management network.
This configuration must be applied to the bootstrap templates,
before you run the bootstrap script to deploy the management
cluster.
Caution
This configuration requires each host in your management
cluster to have at least two physical interfaces.
Connect at least two interfaces per host to an Ethernet switch
that supports Link Aggregation Control Protocol (LACP)
port groups and LACP fallback.
Configure an LACP group on the ports connected
to the NICs of a host.
Configure the LACP fallback on the port group to ensure that
the host can boot over the PXE network before the bond interface
is set up on the host operating system.
Configure server BIOS for both NICs of a bond to be PXE-enabled.
If the server does not support booting from multiple NICs,
configure the port of the LACP group that is connected to the
PXE-enabled NIC of a server to be primary port.
With this setting, the port becomes active in the fallback mode.
To configure a bond interface that aggregates two interfaces
for the PXE/Management network:
In kaas-bootstrap/templates/bm/ipam-objects.yaml.template:
Configure only the following parameters for the declaration
of {{nic0}}, as shown in the example below:
dhcp4
dhcp6
match
set-name
Remove other parameters.
Add the declaration of the second NIC {{nic1}} to be added to the
bond interface:
Specify match:macaddress:{{mac1}} to match the MAC
of the desired NIC.
Specify set-name:{{nic1}} to ensure the correct name of the NIC.
Add the declaration of the bond interface bond0. It must have the
interfaces parameter listing both Ethernet interfaces.
Set the interfaces parameter of the management network bridge
(k8s-lcm in the example below) to include bond0.
Since Container Cloud 2.20.0 and 2.20.1 for MOSK 22.4, each node of every
cluster must have only one IP address in the LCM network that is allocated
from one of the Subnet objects having the ipam/SVC-k8s-lcm label
defined. Therefore, all Subnet objects used for LCM networks must have the
ipam/SVC-k8s-lcm label defined.
Before Container Cloud 2.20.0 and since MOSK 22.2, you can use any interface
name for the LCM network traffic. The Subnet objects for the LCM network
must have the ipam/SVC-k8s-lcm label. For details,
see Service labels and their life cycle.
Set the addresses, gateway4, and nameservers fields of
the management network bridge to fetch data from the kaas-mgmt subnet.
Configure bonding options using the parameters field. The only
mandatory option is mode. See the example below for details.
Note
You can set any mode supported by netplan
and your hardware.
Verify your configuration using the following example:
This section describes the bare metal host profile settings and
instructs how to configure this profile before deploying
Mirantis Container Cloud on physical servers.
The bare metal host profile is a Kubernetes custom resource.
It allows the Infrastructure Operator to define how the storage devices
and the operating system are provisioned and configured.
The bootstrap templates for a bare metal deployment include the template for
the default BareMetalHostProfile object in the following file
that defines the default bare metal host profile:
templates/bm/baremetalhostprofiles.yaml.template
Note
Using BareMetalHostProfile, you can configure LVM or mdadm-based
software RAID support during a management or managed cluster
creation. For details, see Configure RAID support.
This feature is available as Technology Preview. Use such
configuration for testing and evaluation purposes only. For the
Technology Preview feature definition, refer to Technology Preview features.
Warning
All data will be wiped during cluster deployment on devices
defined directly or indirectly in the fileSystems list of
BareMetalHostProfile. For example:
A raw device partition with a file system on it
A device partition in a volume group with a logical volume that has a
file system on it
An mdadm RAID device with a file system on it
An LVM RAID device with a file system on it
The wipe field is always considered true for these devices.
The false value is ignored.
Therefore, to prevent data loss, move the necessary data from these file
systems to another server beforehand, if required.
The customization procedure of BareMetalHostProfile is almost the same for
the management and managed clusters, with the following differences:
For a management cluster, the customization automatically applies
to machines during bootstrap. And for a managed cluster, you apply
the changes using kubectl before creating a managed cluster.
For a management cluster, you edit the default
baremetalhostprofiles.yaml.template. And for a managed cluster, you
create a new BareMetalHostProfile with the necessary configuration.
For the procedure details, see Create a custom bare metal host profile.
Use this procedure for both types of clusters considering the differences
described above.
This section describes how to configure a dedicated PXE network for a
management or regional bare metal cluster.
A separate PXE network allows isolating sensitive bare metal provisioning
process from the end users. The users still have access to Container Cloud
services, such as Keycloak, to authenticate workloads in managed clusters,
such as Horizon in a Mirantis OpenStack for Kubernetes cluster.
The following table describes the overall network mapping scheme with all
L2/L3 parameters, for example, for two networks, PXE (CIDR 10.0.0.0/24)
and management (CIDR 10.0.11.0/24):
When using a separate PXE network, the management cluster services are exposed
in different networks using two separate MetalLB address pools:
Services exposed through the PXE network are as follows:
Ironic API (bare metal provisioning server)
HTTP server that provides images for network boot and server
provisioning
Caching server for accessing the Container Cloud artifacts deployed
on hosts
Services exposed through the management network are all other Container Cloud
services, such as Keycloak, web UI, and so on
To configure separate PXE and management networks:
In kaas-bootstrap/templates/bm/ipam-objects.yaml.template:
Substitute all the Subnet object templates with the new ones
as described in the example template below
Update the L2 template spec.l3Layout and spec.npTemplate fields
as described in the example template below
Example of the Subnet object templates
# Subnet object that provides IP addresses for bare metal hosts of# management cluster in the PXE network.apiVersion:"ipam.mirantis.com/v1alpha1"kind:Subnetmetadata:name:mgmt-pxenamespace:defaultlabels:kaas.mirantis.com/provider:baremetalkaas.mirantis.com/region:region-onekaas-mgmt-pxe-subnet:""spec:cidr:SET_IPAM_CIDRgateway:SET_PXE_NW_GWnameservers:-SET_PXE_NW_DNSincludeRanges:-SET_IPAM_POOL_RANGEexcludeRanges:-SET_METALLB_PXE_ADDR_POOL---# Subnet object that provides IP addresses for bare metal hosts of# management cluster in the management network.apiVersion:"ipam.mirantis.com/v1alpha1"kind:Subnetmetadata:name:mgmt-lcmnamespace:defaultlabels:kaas.mirantis.com/provider:baremetalkaas.mirantis.com/region:region-onekaas-mgmt-lcm-subnet:""ipam/SVC-k8s-lcm:"1"ipam/SVC-ceph-cluster:"1"ipam/SVC-ceph-public:"1"cluster.sigs.k8s.io/cluster-name:CLUSTER_NAMEspec:cidr:{{SET_LCM_CIDR}}includeRanges:-{{SET_LCM_RANGE}}excludeRanges:-SET_LB_HOST-SET_METALLB_ADDR_POOL---# Subnet object that provides configuration for "services-pxe" MetalLB# address pool that will be used to expose services LB endpoints in the# PXE network.apiVersion:"ipam.mirantis.com/v1alpha1"kind:Subnetmetadata:name:mgmt-pxe-lbnamespace:defaultlabels:kaas.mirantis.com/provider:baremetalkaas.mirantis.com/region:region-oneipam/SVC-MetalLB:""metallb/address-pool-name:services-pxemetallb/address-pool-protocol:layer2metallb/address-pool-auto-assign:"false"cluster.sigs.k8s.io/cluster-name:CLUSTER_NAMEspec:cidr:SET_IPAM_CIDRincludeRanges:-SET_METALLB_PXE_ADDR_POOL
The last Subnet template named mgmt-pxe-lb in the example above
will be used to configure the MetalLB address pool in the PXE network.
The bare metal provider will automatically configure MetalLB
with address pools using the Subnet objects identified by specific
labels.
Warning
The bm-pxe address must have a separate interface
with only one address on this interface.
Use the following labels to identify the Subnet object as a MetalLB
address pool and configure the name and protocol for that address pool.
All labels below are mandatory for the Subnet object that configures
a MetalLB address pool.
Mandatory Subnet labels for a MetalLB address pool¶
Label
Description
ipam/SVC-MetalLB
Defines that the Subnet object will be used to provide
a new address pool/range for MetalLB.
metallb/address-pool-name
Sets the name services-pxe for the newly created address pool.
The services-pxe address pool name is mandatory when configuring
a dedicated PXE network in the management cluster. This name will be
used in annotations for services exposed through the PXE network.
Every address pool must have a distinct name except the default
name that is reserved for the management network.
metallb/address-pool-auto-assign
Configures the auto-assign policy of an address pool. Boolean.
Is set to true and is not configurable for address pools
defined through the cluster specification.
For any service that does not have a specific MetalLB annotation
configured, MetalLB allocates external IPs from arbitrary address
pools that have the auto-assign policy set to true.
Only for the service that has a specific MetalLB annotation with the
address pool name, MetalLB allocates external IPs from the address
pool having the auto-assign policy set to false.
metallb/address-pool-protocol
Sets the address pool protocol.
The only supported value is layer2 (default).
cluster.sigs.k8s.io/cluster-name
Specifies the management or regional cluster name that
the Subnet should be bound to.
Caution
Do not set the same address pool name for two or more
Subnet objects. Otherwise, the corresponding MetalLB address pool
configuration fails with a warning message in the bare metal provider
log.
Caution
At least one MetalLB address pool must have the auto-assign
policy enabled so that unannotated services can have load balancer IPs
allocated for them. To satisfy this requirement, either configure one
of address pools using the cluster specification or configure it using
Subnet with metallb/address-pool-auto-assign:"true".
When configuring multiple address pools with the auto-assign
policy enabled, keep in mind that it is not determined which of those
address pools will be used to allocate an IP for a particular
unannotated service.
Verify the current MetalLB configuration:
Since Container Cloud 2.21.0
The MetalLB configuration is stored in MetalLB objects:
The auto-assign parameter will be set to false for all address
pools except the default one. So, a particular service will get an
address from such an address pool only if the Service object has a
special metallb.universe.tf/address-pool annotation that points to
the specific address pool name.
Note
It is expected that every Container Cloud service on a management
and regional cluster will be assigned to one of the address pools.
Current consideration is to have two MetalLB address pools:
services-pxe is a reserved address pool name to use for
the Container Cloud services in the PXE network (Ironic API,
HTTP server, caching server)
default is an address pool to use for all other Container
Cloud services in the management network. No annotation
is required on the Service objects in this case.
In kaas-bootstrap/templates/bm/cluster.yaml.template,
add the dedicatedMetallbPools flag and set it to true:
User sets this flag to enable splitting of LB endpoints for the Container
Cloud services. The metallb.universe.tf/address-pool annotations on the
Service objects are configured by the bare metal provider automatically
when the dedicatedMetallbPools flag is set to true.
Example Service object configured by the baremetal-operator Helm
release:
The metallb.universe.tf/address-pool annotation on the Service
object is set to services-pxe by the baremetal provider, so the
ironic-api service will be assigned an LB address from the
corresponding MetalLB address pool.
Address of a management network for the management cluster
in the CIDR notation. You can later share this network with managed
clusters where it will act as the LCM network.
If managed clusters have their separate LCM networks,
those networks must be routable to the management network.
10.0.11.0/24
SET_LCM_RANGE
Address range that includes addresses to be allocated to
bare metal hosts in the management network for the management
cluster. When this network is shared with managed clusters,
the size of this range limits the number of hosts that can be
deployed in all clusters that share this network.
When this network is solely used by a management cluster,
the range should include at least 3 IP addresses
for bare metal hosts of the management cluster.
10.0.11.100-10.0.11.109
SET_METALLB_PXE_ADDR_POOL
Address range to be used for LB endpoints of the Container Cloud
services: Ironic-API, HTTP server, and caching server.
This range must be within the PXE network.
The minimum required range is 5 IP addresses.
Subnet template parameters migrated to management network¶
Parameter
Description
Example value
SET_LB_HOST
IP address of the externally accessible API endpoint
of the management cluster. This address must NOT be
within the SET_METALLB_ADDR_POOL range but within the
management network. External load balancers are not supported.
10.0.11.90
SET_METALLB_ADDR_POOL
The address range to be used for the externally accessible LB
endpoints of the Container Cloud services, such as Keycloak, web UI,
and so on. This range must be within the management network.
The minimum required range is 19 IP addresses.
Configure multiple DHCP ranges using Subnet resources¶
Caution
Since Container Cloud 2.21.0, this section applies to existing
management clusters only. If you configured multiple DHCP ranges before
Container Cloud 2.21.0 during the management cluster bootstrap, the DHCP
configuration will automatically migrate to Subnet objects after cluster
upgrade to 2.21.0.
To facilitate multi-rack and other types of distributed bare metal datacenter
topologies, the dnsmasq DHCP server used for host provisioning in Container
Cloud supports working with multiple L2 segments through network routers that
support DHCP relay.
Caution
Networks used for hosts provisioning of a managed cluster
must have routes to the PXE network (when a dedicated PXE network
is configured) or to the combined PXE/management network
of the management cluster. This configuration enables hosts to
have access to the management cluster services that are used
during host provisioning.
To configure DHCP ranges for dnsmasq, create the Subnet objects
tagged with the ipam/SVC-dhcp-range label while setting up subnets
for a managed cluster using CLI.
For every dhcp-range record, Container Cloud also configures the
dhcp-option record to pass the default route through the default gateway
from the corresponding subnet to all hosts that obtain addresses
from that DHCP range. They will be configured by Container Cloud using another
dhcp-option record.
Caution
Support of multiple DHCP ranges has the following imitations:
Using of custom DNS server addresses for servers that boot over PXE
is not supported.
The Subnet objects for DHCP ranges should not reference any specific
cluster, as DHCP server configuration is only applicable to the management
or regional cluster. The kaas.mirantis.com/region label that specifies
the region will be used to determine where to apply the DHCP ranges from
the given Subnet object. The Cluster reference will be ignored.
Note
Usage of multiple and single DHCP ranges is as follows:
The baremetal-operator chart allows using multiple DHCP ranges in the
generated dnsmasq.conf file. The chart iterates over a list of the
dhcp-range parameters from its values and adds all items from the list
to the dnsmasq configuration.
The baremetal-operator chart allows using single DHCP range for
backwards compatibility. By default, the KAAS_BM_BM_DHCP_RANGE
environment variable is still used to define the DHCP range for a
management or regional cluster nodes during provisioning.
The dnsmasq configuration options dhcp-option=3 and dhcp-option=6
are absent in the default configuration. So, by default, dnsmasq
will send the DNS server and default route to DHCP clients as defined in the
dnsmasq official documentation:
The netmask and broadcast address are the same as on the host
running dnsmasq.
The DNS server and default route are set to the address of the host
running dnsmasq.
If the domain name option is set, this name is sent to DHCP clients.
If such default behavior is not desirable during deployment of managed
clusters:
Open the management cluster spec for editing.
In the baremetal-operator release values, remove the
dnsmasq.dhcp_range parameter:
The dnsmasq.dhcp_range parameter of the baremetal-operator
Helm chart values in the Clusterspec is deprecated since Container
Cloud 2.21.0 and will be removed in one of the following releases.
Therefore, migrate to the Subnet objects configuration.
Setting of custom nameservers for the PXE subnet is not supported.
After creation of the above Subnet object, the provided data will be
utilized to render the Dnsmasq object used for configuration of the
dnsmasq deployment. You do not have to manually edit the Dnsmasq object.
Verify that the changes are applied to the Dnsmasq object:
For servers to access the DHCP server across the L2 segment boundaries,
for example, from another rack with a different VLAN for PXE network,
you must configure DHCP relay (agent) service on the border switch
of the segment. For example, on a top-of-rack (ToR) or leaf (distribution)
switch, depending on the data center network topology.
Since Container Cloud 2.21.0
The dnsmasq server listens on the PXE network of the management
cluster by using the dhcp-lb Kubernetes Service.
To configure the DHCP relay service, specify the external address of the
dhcp-lb Kubernetes Service as an upstream address for the relayed DHCP
requests, which is the IP helper address for DHCP. There is the dnsmasq
deployment behind this service that can only accept relayed DHCP requests.
Container Cloud has its own DHCP relay running on one of the management
cluster nodes. That DHCP relay serves for proxying DHCP requests in the
same L2 domain where the management cluster nodes are located.
Before Container Cloud 2.21.0
The dnsmasq server listens on the PXE interface of one management
cluster node.
To configure DHCP relay service, specify the management cluster node
addresses in the PXE network as upstream addresses for the relayed DHCP
requests, which are IP helper addresses for DHCP.
Depending on the PXE network setup, select from the following options:
If the PXE network is combined with the management network, identify LCM
addresses of the management cluster nodes:
kubectl-ndefaultgetlcmmachine-owide
In the output, select the addresses from the INTERNALIP column to use
as the DHCP helper addresses.
If you use a dedicated PXE network, identify the addresses assigned
to your nodes using the corresponding IpamHost objects:
kubectl-ndefaultgetipamhost-oyaml
In status.netconfigV2 of each management cluster host, obtain the
interface name used for PXE network and collect associated addresses to use
as the DHCP helper addresses. For example:
In this example, k8s-pxe is the PXE interface name and 10.0.1.4
is the address to use as one of the DHCP helper addresses.
Caution
The following fields of the ipamHost status are renamed since
Container Cloud 2.22.0 in the scope of the L2Template and IpamHost
objects refactoring:
netconfigV2 to netconfigCandidate
netconfigV2state to netconfigCandidateState
netconfigFilesState to netconfigFilesStates (per file)
No user actions are required after renaming.
The format of netconfigFilesState changed after renaming. The
netconfigFilesStates field contains a dictionary of statuses of network
configuration files stored in netconfigFiles. The dictionary contains
the keys that are file paths and values that have the same meaning for each
file that netconfigFilesState had:
For a successfully rendered configuration file:
OK:<timestamp><sha256-hash-of-rendered-file>, where a timestamp
is in the RFC 3339 format.
The system output must contain no error records.
In case of issues, follow the steps provided in Troubleshooting.
Note
If you require all Internet access to go through a proxy server
for security and audit purposes, configure Docker proxy settings
as described in the official
Docker documentation.
After you complete the prerequisite steps described in Prerequisites,
proceed with bootstrapping your OpenStack-based Mirantis Container Cloud
management cluster.
To bootstrap an OpenStack-based management cluster:
Log in to the bootstrap node running Ubuntu 20.04 that is configured
as described in Prerequisites.
Prepare the bootstrap script:
Download and run the Container Cloud bootstrap script:
Log in to your account and download the mirantis.lic license file.
Save the license file as mirantis.lic under the kaas-bootstrap
directory on the bootstrap node.
Verify that mirantis.lic contains the exact Container Cloud license
previously downloaded from www.mirantis.com
by decoding the license JWT token, for example, using jwt.io.
Example of a valid decoded Container Cloud license data with the mandatory
license field:
If you deploy Container Cloud on top of MOSK Victoria with Tungsten Fabric
and use the default security group for newly created load balancers, add the
following rules for the Kubernetes API server endpoint, Container Cloud
application endpoint, and for the MKE web UI and API using the OpenStack CLI:
direction='ingress'
ethertype='IPv4'
protocol='tcp'
remote_ip_prefix='0.0.0.0/0'
port_range_max and port_range_min:
'443' for Kubernetes API and Container Cloud application endpoints
'6443' for MKE web UI and API
Verify access to the target cloud endpoint from Docker. For example:
In case of issues, follow the steps provided in Troubleshooting.
Configure the cluster and machines metadata:
In templates/machines.yaml.template,
modify the spec:providerSpec:value section for 3 control plane nodes
marked with the cluster.sigs.k8s.io/control-plane label
by substituting the flavor and image parameters
with the corresponding values of the control plane nodes in the related
OpenStack cluster. For example:
The flavor parameter value provided in the example above
is cloud-specific and must meet the Container Cloud
requirements.
Also, modify other parameters as required.
Modify the templates/cluster.yaml.template parameters to fit your
deployment. For example, add the corresponding values for cidrBlocks
in the spec::clusterNetwork::services section.
Optional. Available as TechPreview. To boot cluster machines from a block
storage volume, define the following parameter in the spec:providerSpec
section of templates/machines.yaml.template:
To boot the Bastion node from a volume, add the same parameter to
templates/cluster.yaml.template in the spec:providerSpec section
for Bastion. The default amount of storage 80 is enough.
Before Container Cloud 2.23.0, optional if servers from the Ubuntu NTP pool
(*.ubuntu.pool.ntp.org) are accessible from the node where
the management cluster is being provisioned. Otherwise, configure the
regional NTP server parameters as described below.
Since Container Cloud 2.23.0, optionally disable NTP that is enabled by default. This option disables the
management of chrony configuration by Container Cloud to use your own
system for chrony management. Otherwise, configure the regional NTP server
parameters as described below.
NTP configuration
Configure the regional NTP server parameters to be applied to all machines
of regional and managed clusters in the specified region.
In templates/cluster.yaml.template, add the ntp:servers section
with the list of required server names:
Optional. If you require all Internet access to go through a proxy server,
in bootstrap.env, add the following environment variables
to bootstrap the management and regional cluster using proxy:
Optional. Enable infinite timeout for all bootstrap stages by exporting the
following environment variable or adding it to bootstrap.env:
exportKAAS_BOOTSTRAP_INFINITE_TIMEOUT=true
Infinite timeout prevents the bootstrap failure due to timeout. This option
is useful in the following cases:
The network speed is slow for artifacts downloading
An infrastructure configuration does not allow booting fast
A bare-metal node inspecting presupposes more than two HDDSATA disks
to attach to a machine
Optional. Available since Container Cloud 2.23.0. Customize the cluster and
region name by exporting the following environment variables or adding them
to bootstrap.env:
The Keycloak URL that the system outputs when the bootstrap completes.
The admin password for Keycloak is located in
kaas-bootstrap/passwords.yml along with other IAM passwords.
Note
The Container Cloud web UI and StackLight endpoints are available
through Transport Layer Security (TLS) and communicate with Keycloak
to authenticate users. Keycloak is exposed using HTTPS and
self-signed TLS certificates that are not trusted by web browsers.
This section describes how to bootstrap a VMware vSphere-based Mirantis
Container Cloud management cluster.
Note
You can deploy vSphere-based clusters on CentOS. Support of this operating
system is available as Technology Preview.
Use it for testing and evaluation purposes only.
Deployment of a Container Cloud cluster that is based on different operating
systems, such as RHEL and CentOS or CentOS and Ubuntu, is not supported.
The VMware vSphere provider of Mirantis Container Cloud requires the following
resources to successfully create virtual machines for Container Cloud clusters:
Data center
All resources below must be related to one data center.
Cluster
All virtual machines must run on the hosts of one cluster.
Storage for virtual machines disks and Kubernetes volumes.
Folder
Placement of virtual machines.
Resource pool
Pool of CPU and memory resources for virtual machines.
You must provide the data center and cluster resources by name.
You can provide other resources by:
Name
Resource name must be unique in the data center and cluster.
Otherwise, the vSphere provider detects multiple resources with same name
and cannot determine which one to use.
Full path (recommended)
Full path to a resource depends on its type. For example:
The system output must contain no error records.
In case of issues, follow the steps provided in Troubleshooting.
For RHEL:
Log in to a VM running RHEL 7.9 or 8.4 TechPreview that you will
be using as a bootstrap node.
If you do not use RedHat Satellite server locally in your
infrastructure and require all Internet access to
go through a proxy server, including access to RedHat customer
portal, configure proxy parameters for subscription-manager using
the example below:
In MITM proxy deployments, use the internal Red Hat Satellite
server to register RHEL machines so that a VM can access this server directly
without a MITM proxy.
Attach the RHEL subscription using subscription-manager.
Install the following packages:
sudoyuminstallyum-utilswgetvim-y
For RHEL 7.9, verify that the extras repository is enabled:
Add the Docker mirror according to the operating system major version
(7 for 7.9 and 8 for 8.4 TechPreview).
Provide the proxy URL, if required, or set to _none_.
The system output must contain no error records.
In case of issues, follow the steps provided in Troubleshooting.
Note
If you require all Internet access to go through a proxy server
for security and audit purposes, configure Docker proxy settings
as described in the official
Docker documentation.
Prepare the VMware deployment user setup and permissions¶
To deploy Mirantis Container Cloud on the VMware vSphere-based environment,
you need to prepare vSphere accounts for Container Cloud. Contact your vSphere
administrator to set up the required users and permissions following the steps
below:
Log in to the vCenter Server Web Console.
Create the cluster-api user with the following privileges:
Note
Container Cloud uses two separate vSphere accounts for:
Cluster API related operations, such as create or delete VMs,
and for preparation of the VM template using Packer
Storage operations, such as dynamic PVC provisioning
You can also create one user that has all privileges sets
mentioned above.
For RHEL deployments, if you do not have a RHEL machine with the
virt-who service configured to report the vSphere environment
configuration and hypervisors information to RedHat Customer Portal
or RedHat Satellite server, set up the virt-who service
inside the Container Cloud machines for a proper RHEL license activation.
Create a virt-who user with at least read-only access
to all objects in the vCenter Data Center.
The virt-who service on RHEL machines will be provided with the
virt-who user credentials to properly manage RHEL subscriptions.
After you complete the prerequisite steps described in Prerequisites,
proceed with bootstrapping your VMware vSphere-based Mirantis Container Cloud
management cluster.
To bootstrap a vSphere-based management cluster:
Log in to the bootstrap node running Ubuntu 20.04 that is configured
as described in Prerequisites.
Prepare the bootstrap script:
Download and run the Container Cloud bootstrap script:
Log in to your account and download the mirantis.lic license file.
Save the license file as mirantis.lic under the kaas-bootstrap
directory on the bootstrap node.
Verify that mirantis.lic contains the exact Container Cloud license
previously downloaded from www.mirantis.com
by decoding the license JWT token, for example, using jwt.io.
Example of a valid decoded Container Cloud license data with the mandatory
license field:
IP address from the provided vSphere network for load balancer (Keepalived).
SET_VSPHERE_METALLB_RANGE
MetalLB range of IP addresses that can be assigned to load balancers
for Kubernetes Services.
SET_VSPHERE_DATASTORE
Name of the vSphere datastore. You can use different datastores
for vSphere Cluster API and vSphere Cloud Provider.
SET_VSPHERE_MACHINES_FOLDER
Path to a folder where the cluster machines metadata will be stored.
SET_VSPHERE_NETWORK_PATH
Path to a network for cluster machines.
SET_VSPHERE_RESOURCE_POOL_PATH
Path to a resource pool in which VMs will be created.
Note
To obtain the LB_HOST and VSPHERE_METALLB_RANGE
parameters for the selected vSphere network, contact your vSphere
administrator who provides you with IP ranges dedicated to your
environment only.
Modify other parameters if required. For example, add the corresponding
values for cidrBlocks in the spec::clusterNetwork::services section.
Provide the following additional parameters for a proper network setup
on machines using embedded IP address management (IPAM)
in templates/vsphere/cluster.yaml.template
Note
To obtain IPAM parameters for the selected vSphere network, contact
your vSphere administrator who provides you with IP ranges dedicated to your
environment only.
Enables IPAM. Recommended value is true for either DHCP or non-DHCP networks.
SET_VSPHERE_NETWORK_CIDR
CIDR of the provided vSphere network. For example, 10.20.0.0/16.
SET_VSPHERE_NETWORK_GATEWAY
Gateway of the provided vSphere network.
SET_VSPHERE_CIDR_INCLUDE_RANGES
IP range for the cluster machines. Specify the range of the provided CIDR.
For example, 10.20.0.100-10.20.0.200. If the DHCP network is used, this
range must not intersect with the DHCP range of the network.
SET_VSPHERE_CIDR_EXCLUDE_RANGES
Optional. IP ranges to be excluded from being assigned to the cluster
machines. The MetalLB range and SET_LB_HOST should not intersect with
the addresses for IPAM. For example, 10.20.0.150-10.20.0.170.
SET_VSPHERE_NETWORK_NAMESERVERS
List of nameservers for the provided vSphere network.
For RHEL deployments, fill out
templates/vsphere/rhellicenses.yaml.template
using one of the following set of parameters for RHEL machines subscription:
The user name and password of your RedHat Customer Portal account
associated with your RHEL license for Virtual Datacenters.
Optionally, provide the subscription allocation pools to use for the RHEL
subscription activation. If not needed, remove the poolIDs field
for subscription-manager to automatically select the licenses for
machines.
The activation key and organization ID associated with your RedHat
account with RHEL license for Virtual Datacenters. The activation key can
be created by the organization administrator on the RedHat Customer Portal.
If you use the RedHat Satellite server for management of your
RHEL infrastructure, you can provide a pre-generated activation key from
that server. In this case:
Provide the URL to the RedHat Satellite RPM for installation
of the CA certificate that belongs to that server.
Configure squid-proxy on the management or regional cluster to allow
access to your Satellite server. For details, see Configure squid-proxy.
For RHEL 8.4 TechPreview, verify mirrors configuration
for your activation key. For more details, see RHEL 8 mirrors configuration.
Warning
Provide only one set of parameters. Mixing the parameters from
different activation methods will cause deployment failure.
For CentOS deployments, in templates/vsphere/rhellicenses.yaml.template,
remove all lines under items:.
In bootstrap.env, add the KAAS_VSPHERE_ENABLED=true environment
variable that enables the vSphere provider deployment in Container Cloud.
Configure NTP server.
Before Container Cloud 2.23.0, optional if servers from the Ubuntu NTP pool
(*.ubuntu.pool.ntp.org) are accessible from the node where
the management cluster is being provisioned. Otherwise, configure the
regional NTP server parameters as described below.
Since Container Cloud 2.23.0, optionally disable NTP that is enabled by default. This option disables the
management of chrony configuration by Container Cloud to use your own
system for chrony management. Otherwise, configure the regional NTP server
parameters as described below.
NTP configuration
Configure the regional NTP server parameters to be applied to all machines
of regional and managed clusters in the specified region.
In templates/vsphere/cluster.yaml.template, add the ntp:servers section
with the list of required server names:
In templates/vsphere/machines.yaml.template, define the following
parameters:
rhelLicense
RHEL license name defined in rhellicenses.yaml.template, defaults to
kaas-mgmt-rhel-license. Remove or comment out this parameter for
CentOS and Ubuntu deployments.
diskGiB
Disk size in GiB for machines that must match the disk size of the VM
template. You can leave this parameter commented to use the disk size of
the VM template. The minimum requirement is 120 GiB.
template
Path to the VM template prepared in the previous step.
If you require all Internet access to go through a proxy server,
in bootstrap.env, add the following environment variables
to bootstrap the management and regional cluster using proxy:
In MITM proxy deployments, use the internal Red Hat Satellite
server to register RHEL machines so that a VM can access this server directly
without a MITM proxy.
Optional. Enable infinite timeout for all bootstrap stages by exporting the
following environment variable or adding it to bootstrap.env:
exportKAAS_BOOTSTRAP_INFINITE_TIMEOUT=true
Infinite timeout prevents the bootstrap failure due to timeout. This option
is useful in the following cases:
The network speed is slow for artifacts downloading
An infrastructure configuration does not allow booting fast
A bare-metal node inspecting presupposes more than two HDDSATA disks
to attach to a machine
Optional. Available since Container Cloud 2.23.0. Customize the cluster and
region name by exporting the following environment variables or adding them
to bootstrap.env:
The Keycloak URL that the system outputs when the bootstrap completes.
The admin password for Keycloak is located in
kaas-bootstrap/passwords.yml along with other IAM passwords.
Note
The Container Cloud web UI and StackLight endpoints are available
through Transport Layer Security (TLS) and communicate with Keycloak
to authenticate users. Keycloak is exposed using HTTPS and
self-signed TLS certificates that are not trusted by web browsers.
To deploy Mirantis Container Cloud on the vSphere-based environment, prepare
the virtual machine (VM) template for cluster machines that fits the following
requirements:
The VMware Tools package is installed.
The cloud-init utility is installed and configured with the
specific VMwareGuestInfo data source.
The following procedures describe how to meet the requirements above
either using the Container Cloud script or manually.
Prepare the VM template using the Container Cloud script¶
Prepare the Container Cloud bootstrap and modify
templates/vsphere/vsphere-config.yaml.template and
templates/vsphere/cluster.yaml.template
as described in Bootstrap a management cluster, steps 1-9.
Download or add to the vSphere datastore the ISO image depending on the
target operating system:
Ubuntu 20.04 Server Install Image from Ubuntu images
Technology Preview: CentOS 7.9 DVD ISO from the CentOS mirrors
Export the environment variable for the ISO file depending on its placement:
# On the seed nodeexportVSPHERE_PACKER_ISO_FILE=$(pwd)/iso-file.dvd.iso
# On the vSphere datastoreexportVSPHERE_PACKER_STORAGE_PATH="[<datastoreName>] /<path/to>/iso-file.dvd.iso"
Verify that the Docker service is running and the bootstrap node user is
added to the docker group.
For RHEL, SELinux has to be in permissive mode or disabled.
For more details about the bootstrap seed node prerequisites, see
Prerequisites.
Export the following variables:
The path to the downloaded ISO file.
The vSphere cluster name.
The OS name: rhel, ubuntu, or centos.
The OS version: 7.8, 7.9, or 8.4 (Technology Preview)
for RHEL;
7.9 for CentOS (Technology Preview), 20.04 for Ubuntu.
In MITM proxy deployments, use the internal Red Hat Satellite
server to register RHEL machines so that a VM can access this server directly
without a MITM proxy.
After the template is prepared, set the <vSphereVMTemplatePath>
parameter in templates/vsphere/machines.yaml.template as described
in Bootstrap a management cluster.
Add 99-DataSourceVMwareGuestInfo.cfg to /etc/cloud/cloud.cfg.d/.
Depending on the Python version on the VM operating system,
add DataSourceVMwareGuestInfo.py to the cloud-init sources
folder. Obtain the cloud-init folder on the OS:
python-c'import os; from cloudinit import sources; print(os.path.dirname(sources.__file__));'
For Ubuntu, create /etc/cloud/cloud.cfg.d/99_mcc.cfg with the
following content:
For CentOS, verify that .yum mirrors are set to use only the
*.centos.org URLs. Otherwise, access to other mirrors may be blocked by
squid-proxy on managed clusters. Refer to Configure squid-proxy on
how to allow access to custom mirrors.
Configure the interface name for eth0:
In /etc/default/grub, add the following parameters to
GRUB_CMDLINE_LINUX:
By default squid-proxy allows an access only to the official RedHat
subscription.rhsm.redhat.com and .cdn.redhat.com URLs or to the
CentOS *.centos.org mirrors.
If you use RedHat Satellite server or if you want to access some specific
yum repositories of RedHat or CentOS, allow those domains
(or IPs addresses) in the squid-proxy configuration
on the management or regional cluster.
Note
You can apply the procedure below before or after the management or
regional cluster deployment.
To configure squid-proxy for an access to specific domains:
Modify the allowed domains for squid-proxy in the regional Helm
releases configuration for the vsphere provider using the example below.
For new deployments, modify templates/vsphere/cluster.yaml.template
For existing deployments, modify the management or regional cluster
configuration:
By default, the RHEL subscription grants access to the AppStream
and BaseOS repositories that are not bound to a specific operating system
version and that are stream repositories, so they are frequently updated.
To deploy RHEL 8.4 and make sure that packages are installed
from the version 8.4 AppStream and BaseOS repositories,
RHEL VM template have the releasever variable for .yum set to 8.4.
You can verify this variable in /etc/yum/vars/releasever on a VM.
If you are using the RedHat Satellite server, verify that your activation key
is configured with the release version set to 8.4 and includes only
the following repositories:
Red Hat Enterprise Linux 8 for x86_64 - BaseOS RPMs 8.4
Red Hat Enterprise Linux 8 for x86_64 - AppStream RPMs 8.4
After you bootstrap a management cluster of the required cloud provider type,
you can optionally deploy an additional regional cluster of the same
or different provider type. For details about regions, see Container Cloud regions.
Perform this procedure if you wish to operate managed clusters across clouds
from a single Mirantis Container Cloud management plane.
Caution
A regional cluster requires access to the management cluster.
Multi-regional deployment enables you to create managed clusters of several
provider types using one management cluster. For example, you can bootstrap a
vSphere-based management cluster and deploy an OpenStack-based regional
cluster on this management cluster. Such cluster enables creation of
OpenStack-based and vSphere-based managed clusters with Kubernetes deployments.
Note
If the bootstrap node for deployment of an additional regional
cluster is not the same where you bootstrapped the management
cluster, first prepare the bootstrap as described in
Configure the bootstrap node.
This section describes how to prepare a new bootstrap node for an additional
regional cluster deployment on top of the management cluster.
To use the same node where you bootstrapped the management cluster, skip this
instruction and proceed to deploying a regional cluster of the required
provider type.
To configure a new bootstrap node for a regional cluster:
Install and configure Docker:
Log in to any personal computer or VM running Ubuntu 20.04
that you will be using as the bootstrap node.
If you use a newly created VM, run:
sudoapt-getupdate
Install the current Docker version available for Ubuntu 20.04:
sudoaptinstalldocker.io
Grant your USER access to the Docker daemon:
sudousermod-aGdocker$USER
Log off and log in again to the bootstrap node to apply the changes.
Verify that Docker is configured correctly and has access
to Container Cloud CDN. For example:
The system output must contain no error records.
In case of issues, follow the steps provided in Troubleshooting.
Note
If you require all Internet access to go through a proxy server
for security and audit purposes, configure Docker proxy settings
as described in the official
Docker documentation.
Prepare the bootstrap script:
Download and run the Container Cloud bootstrap script:
Log in to your account and download the mirantis.lic license file.
Save the license file as mirantis.lic under the kaas-bootstrap
directory on the bootstrap node.
Verify that mirantis.lic contains the exact Container Cloud license
previously downloaded from www.mirantis.com
by decoding the license JWT token, for example, using jwt.io.
Example of a valid decoded Container Cloud license data with the mandatory
license field:
For clusters deployed using the Container Cloud release earlier than 2.11.0
or if you deleted the kaas-bootstrap folder, download and run
the Container Cloud bootstrap script:
If you deploy Container Cloud on top of MOSK Victoria with Tungsten Fabric
and use the default security group for newly created load balancers, add the
following rules for the Kubernetes API server endpoint, Container Cloud
application endpoint, and for the MKE web UI and API using the OpenStack CLI:
direction='ingress'
ethertype='IPv4'
protocol='tcp'
remote_ip_prefix='0.0.0.0/0'
port_range_max and port_range_min:
'443' for Kubernetes API and Container Cloud application endpoints
'6443' for MKE web UI and API
Verify access to the target cloud endpoint from Docker. For example:
In case of issues, follow the steps provided in Troubleshooting.
Configure the cluster and machines metadata:
In templates/machines.yaml.template,
modify the spec:providerSpec:value section for 3 control plane nodes
marked with the cluster.sigs.k8s.io/control-plane label
by substituting the flavor and image parameters
with the corresponding values of the control plane nodes in the related
OpenStack cluster. For example:
The flavor parameter value provided in the example above
is cloud-specific and must meet the Container Cloud
requirements.
Also, modify other parameters as required.
Modify the templates/cluster.yaml.template parameters to fit your
deployment. For example, add the corresponding values for cidrBlocks
in the spec::clusterNetwork::services section.
Optional. Available as TechPreview. To boot cluster machines from a block
storage volume, define the following parameter in the spec:providerSpec
section of templates/machines.yaml.template:
To boot the Bastion node from a volume, add the same parameter to
templates/cluster.yaml.template in the spec:providerSpec section
for Bastion. The default amount of storage 80 is enough.
Configure NTP server.
Before Container Cloud 2.23.0, optional if servers from the Ubuntu NTP pool
(*.ubuntu.pool.ntp.org) are accessible from the node where
the regional cluster is being provisioned. Otherwise, configure the
regional NTP server parameters as described below.
Since Container Cloud 2.23.0, optionally disable NTP that is enabled by default. This option disables the
management of chrony configuration by Container Cloud to use your own
system for chrony management. Otherwise, configure the regional NTP server
parameters as described below.
NTP configuration
Configure the regional NTP server parameters to be applied to all machines
of regional and managed clusters in the specified region.
In templates/cluster.yaml.template, add the ntp:servers section
with the list of required server names:
Optional. If you require all Internet access to go through a proxy server,
in bootstrap.env, add the following environment variables
to bootstrap the regional cluster using proxy:
Substitute the parameters enclosed in angle brackets with the corresponding
values of your cluster.
Caution
The REGION and REGIONAL_CLUSTER_NAME parameters values
must contain only lowercase alphanumeric characters, hyphens,
or periods.
Note
If the bootstrap node for the regional cluster deployment is not
the same where you bootstrapped the management cluster, also
export SSH_KEY_NAME. It is required for the management
cluster to create a publicKey Kubernetes CRD with the
public part of your newly generated ssh_key
for the regional cluster.
exportSSH_KEY_NAME=<newRegionalClusterSshKeyName>
Run the regional cluster bootstrap script:
./bootstrap.shdeploy_regional
Note
When the bootstrap is complete, obtain and save in a secure location
the kubeconfig-<regionalClusterName> file
located in the same directory as the bootstrap script.
This file contains the admin credentials for the regional cluster.
If the bootstrap node for the regional cluster deployment is not
the same where you bootstrapped the management cluster, a new
regional ssh_key will be generated.
Make sure to save this key in a secure location as well.
The workflow of the regional cluster bootstrap script¶
#
Description
1
Prepare the bootstrap cluster for the new regional cluster.
2
Load the updated Container Cloud CRDs for Credentials,
Cluster, and Machines with information about the new
regional cluster to the management cluster.
3
Connect to each machine of the management cluster through SSH.
4
Wait for the Machines and Cluster objects of the new regional
cluster to be ready on the management cluster.
5
Load the following objects to the new regional cluster: Secret
with the management cluster kubeconfig and ClusterRole for
the Container Cloud provider.
6
Forward the bootstrap cluster endpoint to helm-controller.
7
Wait for all CRDs to be available and verify the objects
created using these CRDs.
8
Pivot the cluster API stack to the regional cluster.
9
Switch the LCM Agent from the bootstrap cluster to the regional one.
10
Wait for the Container Cloud components to start on the regional
cluster.
You can deploy an additional regional baremetal-based cluster
to create managed clusters of several provider types or with different
configurations within a single Container Cloud deployment.
To deploy a baremetal-based regional cluster:
Log in to the node where you bootstrapped the Container Cloud management
cluster.
Verify that the bootstrap directory is updated.
Select from the following options:
For clusters deployed using Container Cloud 2.11.0 or later:
For clusters deployed using the Container Cloud release earlier than 2.11.0
or if you deleted the kaas-bootstrap folder, download and run
the Container Cloud bootstrap script:
Prepare the bare metal configuration for the new regional cluster:
Create a virtual bridge to connect to your PXE network on the
seed node. Use the following netplan-based configuration file
as an example:
# cat /etc/netplan/config.yamlnetwork:version:2renderer:networkdethernets:ens3:dhcp4:falsedhcp6:falsebridges:br0:addresses:# Please, adjust for your environment-10.0.0.15/24dhcp4:falsedhcp6:false# Please, adjust for your environmentgateway4:10.0.0.1interfaces:# Interface name may be different in your environment-ens3nameservers:addresses:# Please, adjust for your environment-8.8.8.8parameters:forward-delay:4stp:false
Apply the new network configuration using netplan:
The system output must contain a json file with no error messages.
In case of errors, follow the steps provided in Troubleshooting.
Note
If you require all Internet access to go through a proxy server
for security and audit purposes, configure Docker proxy settings
as described in the official
Docker documentation.
Prepare the deployment configuration files that contain the cluster
and machines metadata:
Create a copy of the current templates directory for future reference.
Update the cluster definition template in
templates/bm/cluster.yaml.template
according to the environment configuration. Use the table below.
Manually set all parameters that start with SET_. For example,
SET_METALLB_ADDR_POOL.
The IP address of the externally accessible API endpoint
of the cluster. This address must NOT be
within the SET_METALLB_ADDR_POOL range but must be within
the PXE/Management network. External load balancers are not supported.
10.0.0.90
SET_METALLB_ADDR_POOL
The IP range to be used as external load balancers for the Kubernetes
services with the LoadBalancer type. This range must be within
the PXE/Management network. The minimum required range is 19 IP addresses.
10.0.0.61-10.0.0.80
Configure NTP server.
Before Container Cloud 2.23.0, optional if servers from the Ubuntu NTP pool
(*.ubuntu.pool.ntp.org) are accessible from the node where
your cluster is being provisioned. Otherwise, configure the
regional NTP server parameters as described below.
Since Container Cloud 2.23.0, optionally disable NTP that is enabled by default. This option disables the
management of chrony configuration by Container Cloud to use your own
system for chrony management. Otherwise, configure the regional NTP server
parameters as described below.
NTP configuration
Configure the regional NTP server parameters to be applied to all machines
of regional and managed clusters in the specified region.
In templates/bm/cluster.yaml.template, add the ntp:servers section
with the list of required server names:
Inspect the default bare metal host profile definition in
templates/bm/baremetalhostprofiles.yaml.template.
If your hardware configuration differs from the reference,
adjust the default profile to match. For details, see
Customize the default bare metal host profile.
Warning
All data will be wiped during cluster deployment on devices
defined directly or indirectly in the fileSystems list of
BareMetalHostProfile. For example:
A raw device partition with a file system on it
A device partition in a volume group with a logical volume that has a
file system on it
An mdadm RAID device with a file system on it
An LVM RAID device with a file system on it
The wipe field is always considered true for these devices.
The false value is ignored.
Therefore, to prevent data loss, move the necessary data from these file
systems to another server beforehand, if required.
Update the bare metal hosts definition template in
templates/bm/baremetalhosts.yaml.template
according to the environment configuration. Use the table below.
Manually set all parameters that start with SET_.
The MAC address of the first master node in the PXE network.
ac:1f:6b:02:84:71
SET_MACHINE_0_BMC_ADDRESS
The IP address of the BMC endpoint for the first master node in
the cluster. Must be an address from the OOB network
that is accessible through the PXE network default gateway.
The MAC address of the second master node in the PXE network.
ac:1f:6b:02:84:72
SET_MACHINE_1_BMC_ADDRESS
The IP address of the BMC endpoint for the second master node in
the cluster. Must be an address from the OOB network
that is accessible through the PXE network default gateway.
The MAC address of the third master node in the PXE network.
ac:1f:6b:02:84:73
SET_MACHINE_2_BMC_ADDRESS
The IP address of the BMC endpoint for the third master node in
the cluster. Must be an address from the OOB network
that is accessible through the PXE network default gateway.
Since Container Cloud 2.21.0, a user name and password in plain
text are required.
Before Container Cloud 2.21.0, the Base64 encoding of a user name
and password is required. You can obtain the Base64-encoded user
name and password using the following command in your Linux
console:
$echo-n<username|password>|base64
Update the Subnet objects definition template in
templates/bm/ipam-objects.yaml.template
according to the environment configuration. Use the table below.
Manually set all parameters that start with SET_.
For example, SET_IPAM_POOL_RANGE.
The IP address of the externally accessible API endpoint
of the cluster. This address must NOT be
within the SET_METALLB_ADDR_POOL range but must be within the
PXE/Management network. External load balancers are not supported.
The IP address range to be used as external load balancers for the
Kubernetes services with the LoadBalancer type. This range must
be within the PXE/Management network. The minimum required range is
19 IP addresses.
Use the same value that you used for this parameter in the
cluster.yaml.template file (see above).
Optional. To configure the separated PXE and management networks instead of
one PXE/management network, proceed to Separate PXE and management networks.
Optional. To connect the cluster hosts to the PXE/Management
network using bond interfaces, proceed to Configure NIC bonding.
If you require all Internet access to go through a proxy server,
in bootstrap.env, add the following environment variables to bootstrap
the cluster using proxy:
The provisioning IP address. This address will be assigned to the
interface of the seed node defined by the KAAS_BM_PXE_BRIDGE
parameter (see below). The PXE service of the bootstrap cluster will
use this address to network boot the bare metal hosts for the
cluster.
10.0.0.20
KAAS_BM_PXE_MASK
The CIDR prefix for the PXE network. It will be used with KAAS_BM_PXE_IP
address when assigning it to network interface.
24
KAAS_BM_PXE_BRIDGE
The PXE network bridge name. The name must match the name
of the bridge created on the seed node during the
Prepare the seed node stage.
br0
KAAS_BM_BM_DHCP_RANGE
The start_ip and end_ip addresses must be within the PXE network.
This range will be used by dnsmasq to provide IP addresses for nodes
during provisioning.
10.0.0.30,10.0.0.49,255.255.255.0
BOOTSTRAP_METALLB_ADDRESS_POOL
The pool of IP addresses that will be used by services
in the bootstrap cluster. Can be the same as the
SET_METALLB_ADDR_POOL range for the cluster, or a different range.
10.0.0.61-10.0.0.80
Run the verification preflight script to validate the deployment
templates configuration:
./bootstrap.shpreflight
The command outputs a human-readable report with the verification details.
The report includes the list of verified bare metal nodes and their
ChassisPower status.
This status is based on the deployment templates configuration used
during the verification.
Caution
If the report contains information about missing dependencies
or incorrect configuration, fix the issues before proceeding
to the next step.
Verify that the vSphere provider selection parameter is unset:
Substitute the parameters enclosed in angle brackets with the corresponding
values of your cluster.
Caution
The REGION and REGIONAL_CLUSTER_NAME parameters values
must contain only lowercase alphanumeric characters, hyphens,
or periods.
Note
If the bootstrap node for the regional cluster deployment is not
the same where you bootstrapped the management cluster, also
export SSH_KEY_NAME. It is required for the management
cluster to create a publicKey Kubernetes CRD with the
public part of your newly generated ssh_key
for the regional cluster.
exportSSH_KEY_NAME=<newRegionalClusterSshKeyName>
Run the regional cluster bootstrap script:
./bootstrap.shdeploy_regional
Note
When the bootstrap is complete, obtain and save in a secure location
the kubeconfig-<regionalClusterName> file
located in the same directory as the bootstrap script.
This file contains the admin credentials for the regional cluster.
If the bootstrap node for the regional cluster deployment is not
the same where you bootstrapped the management cluster, a new
regional ssh_key will be generated.
Make sure to save this key in a secure location as well.
The workflow of the regional cluster bootstrap script¶
#
Description
1
Prepare the bootstrap cluster for the new regional cluster.
2
Load the updated Container Cloud CRDs for Credentials,
Cluster, and Machines with information about the new
regional cluster to the management cluster.
3
Connect to each machine of the management cluster through SSH.
4
Wait for the Machines and Cluster objects of the new regional
cluster to be ready on the management cluster.
5
Load the following objects to the new regional cluster: Secret
with the management cluster kubeconfig and ClusterRole for
the Container Cloud provider.
6
Forward the bootstrap cluster endpoint to helm-controller.
7
Wait for all CRDs to be available and verify the objects
created using these CRDs.
8
Pivot the cluster API stack to the regional cluster.
9
Switch the LCM Agent from the bootstrap cluster to the regional one.
10
Wait for the Container Cloud components to start on the regional
cluster.
You can deploy an additional regional VMware vSphere-based cluster
to create managed clusters of several provider types or with different
configurations.
To deploy a vSphere-based regional cluster:
Log in to the node where you bootstrapped a management
cluster.
Verify that the bootstrap directory is updated.
Select from the following options:
For clusters deployed using Container Cloud 2.11.0 or later:
For clusters deployed using the Container Cloud release earlier than 2.11.0
or if you deleted the kaas-bootstrap folder, download and run
the Container Cloud bootstrap script:
IP address from the provided vSphere network for load balancer (Keepalived).
SET_VSPHERE_METALLB_RANGE
MetalLB range of IP addresses that can be assigned to load balancers
for Kubernetes Services.
SET_VSPHERE_DATASTORE
Name of the vSphere datastore. You can use different datastores
for vSphere Cluster API and vSphere Cloud Provider.
SET_VSPHERE_MACHINES_FOLDER
Path to a folder where the cluster machines metadata will be stored.
SET_VSPHERE_NETWORK_PATH
Path to a network for cluster machines.
SET_VSPHERE_RESOURCE_POOL_PATH
Path to a resource pool in which VMs will be created.
Note
To obtain the LB_HOST and VSPHERE_METALLB_RANGE
parameters for the selected vSphere network, contact your vSphere
administrator who provides you with IP ranges dedicated to your
environment only.
Modify other parameters if required. For example, add the corresponding
values for cidrBlocks in the spec::clusterNetwork::services section.
Provide the following additional parameters for a proper network setup
on machines using embedded IP address management (IPAM)
in templates/vsphere/cluster.yaml.template
Note
To obtain IPAM parameters for the selected vSphere network, contact
your vSphere administrator who provides you with IP ranges dedicated to your
environment only.
Enables IPAM. Recommended value is true for either DHCP or non-DHCP networks.
SET_VSPHERE_NETWORK_CIDR
CIDR of the provided vSphere network. For example, 10.20.0.0/16.
SET_VSPHERE_NETWORK_GATEWAY
Gateway of the provided vSphere network.
SET_VSPHERE_CIDR_INCLUDE_RANGES
IP range for the cluster machines. Specify the range of the provided CIDR.
For example, 10.20.0.100-10.20.0.200. If the DHCP network is used, this
range must not intersect with the DHCP range of the network.
SET_VSPHERE_CIDR_EXCLUDE_RANGES
Optional. IP ranges to be excluded from being assigned to the cluster
machines. The MetalLB range and SET_LB_HOST should not intersect with
the addresses for IPAM. For example, 10.20.0.150-10.20.0.170.
SET_VSPHERE_NETWORK_NAMESERVERS
List of nameservers for the provided vSphere network.
For RHEL deployments, fill out
templates/vsphere/rhellicenses.yaml.template
using one of the following set of parameters for RHEL machines subscription:
The user name and password of your RedHat Customer Portal account
associated with your RHEL license for Virtual Datacenters.
Optionally, provide the subscription allocation pools to use for the RHEL
subscription activation. If not needed, remove the poolIDs field
for subscription-manager to automatically select the licenses for
machines.
The activation key and organization ID associated with your RedHat
account with RHEL license for Virtual Datacenters. The activation key can
be created by the organization administrator on the RedHat Customer Portal.
If you use the RedHat Satellite server for management of your
RHEL infrastructure, you can provide a pre-generated activation key from
that server. In this case:
Provide the URL to the RedHat Satellite RPM for installation
of the CA certificate that belongs to that server.
Configure squid-proxy on the management or regional cluster to allow
access to your Satellite server. For details, see Configure squid-proxy.
For RHEL 8.4 TechPreview, verify mirrors configuration
for your activation key. For more details, see RHEL 8 mirrors configuration.
Warning
Provide only one set of parameters. Mixing the parameters from
different activation methods will cause deployment failure.
For CentOS deployments, in templates/vsphere/rhellicenses.yaml.template,
remove all lines under items:.
Configure NTP server.
Before Container Cloud 2.23.0, optional if servers from the Ubuntu NTP pool
(*.ubuntu.pool.ntp.org) are accessible from the node where
the regional cluster is being provisioned. Otherwise, configure the
regional NTP server parameters as described below.
Since Container Cloud 2.23.0, optionally disable NTP that is enabled by default. This option disables the
management of chrony configuration by Container Cloud to use your own
system for chrony management. Otherwise, configure the regional NTP server
parameters as described below.
NTP configuration
Configure the regional NTP server parameters to be applied to all machines
of regional and managed clusters in the specified region.
In templates/vsphere/cluster.yaml.template, add the ntp:servers section
with the list of required server names:
In templates/vsphere/machines.yaml.template, define the following
parameters:
rhelLicense
RHEL license name defined in rhellicenses.yaml.template, defaults to
kaas-mgmt-rhel-license. Remove or comment out this parameter for
CentOS and Ubuntu deployments.
diskGiB
Disk size in GiB for machines that must match the disk size of the VM
template. You can leave this parameter commented to use the disk size of
the VM template. The minimum requirement is 120 GiB.
template
Path to the VM template prepared in the previous step.
Optional. If you require all Internet access to go through a proxy server,
in bootstrap.env, add the following environment variables
to bootstrap the regional cluster using proxy:
Substitute the parameters enclosed in angle brackets with the corresponding
values of your cluster.
Caution
The REGION and REGIONAL_CLUSTER_NAME parameters values
must contain only lowercase alphanumeric characters, hyphens,
or periods.
Note
If the bootstrap node for the regional cluster deployment is not
the same where you bootstrapped the management cluster, also
export SSH_KEY_NAME. It is required for the management
cluster to create a publicKey Kubernetes CRD with the
public part of your newly generated ssh_key
for the regional cluster.
exportSSH_KEY_NAME=<newRegionalClusterSshKeyName>
Run the regional cluster bootstrap script:
./bootstrap.shdeploy_regional
Note
When the bootstrap is complete, obtain and save in a secure location
the kubeconfig-<regionalClusterName> file
located in the same directory as the bootstrap script.
This file contains the admin credentials for the regional cluster.
If the bootstrap node for the regional cluster deployment is not
the same where you bootstrapped the management cluster, a new
regional ssh_key will be generated.
Make sure to save this key in a secure location as well.
The workflow of the regional cluster bootstrap script¶
#
Description
1
Prepare the bootstrap cluster for the new regional cluster.
2
Load the updated Container Cloud CRDs for Credentials,
Cluster, and Machines with information about the new
regional cluster to the management cluster.
3
Connect to each machine of the management cluster through SSH.
4
Wait for the Machines and Cluster objects of the new regional
cluster to be ready on the management cluster.
5
Load the following objects to the new regional cluster: Secret
with the management cluster kubeconfig and ClusterRole for
the Container Cloud provider.
6
Forward the bootstrap cluster endpoint to helm-controller.
7
Wait for all CRDs to be available and verify the objects
created using these CRDs.
8
Pivot the cluster API stack to the regional cluster.
9
Switch the LCM Agent from the bootstrap cluster to the regional one.
10
Wait for the Container Cloud components to start on the regional
cluster.
For MOSK, the feature is generally available since
MOSK 23.1.
While bootstrapping a Container Cloud management or regional cluster using
proxy, you may require Internet access to go through a man-in-the-middle
(MITM) proxy. Such configuration requires that you enable streaming and
install a CA certificate on a bootstrap node.
Replace ~/.mitmproxy/mitmproxy-ca-cert.cer with the path to your CA
certificate.
Caution
The target CA certificate file must be in the PEM format
with the .crt extension.
Apply the changes:
sudoupdate-ca-certificates
Now, proceed with bootstrapping your management or regional cluster.
Create initial users after a management cluster bootstrap¶
Once you bootstrap your management or regional cluster,
create Keycloak users for access to the Container Cloud web UI.
Use the created credentials to log in to the Container Cloud web UI.
Mirantis recommends creating at least two users, user and operator,
that are required for a typical Container Cloud deployment.
To create the user for access to the Container Cloud web UI, use the following
command:
You will be asked for the user password interactively.
Set the following command flags as required:
Flag
Description
--username
Required. Name of the user to create.
--roles
Required. Comma-separated roles to assign to the user.
If you run the command without the --namespace flag,
you can assign the following roles:
global-admin - read and write access for global role bindings
writer - read and write access
reader - view access
operator - required for bare metal deployments only
to create and manage the BaremetalHost objects
If you run the command for a specific project using the --namespace
flag, you can assign the following roles:
operator or writer - read and write access
user or reader - view access
member - read and write access (excluding IAM objects)
bm-pool-operator - required for bare metal deployments only
to create and manage the BaremetalHost objects
--kubeconfig
Required. Path to the management cluster kubeconfig generated during
the management cluster bootstrap.
--namespace
Optional. Name of the Container Cloud project where the user will be
created. If not set, a global user will be created for all Container
Cloud projects with the corresponding role access to view or manage
all Container Cloud public objects.
--password-stdin
Optional. Flag to provide the user password from a file or stdin:
For clusters deployed using the Container Cloud release earlier than 2.11.0
or if you deleted the kaas-bootstrap folder, download and run
the Container Cloud bootstrap script:
Add COLLECT_EXTENDED_LOGS=true before the command to output the
extended version of logs that contains system and MKE logs, logs from
LCM Ansible and LCM Agent along with cluster events and Kubernetes
resources description and logs.
Without the --extended flag, the basic version of logs is collected, which
is sufficient for most use cases. The basic version of logs contains all
events, Kubernetes custom resources, and logs from all Container Cloud
components. This version does not require passing --key-file.
The logs are collected in the directory where the bootstrap script
is located.
Technology Preview. For bare metal clusters, assess the Ironic pod logs:
Extract the content of the 'message' fields from every log message:
/objects/cluster - logs of the non-namespaced Kubernetes objects.
/objects/namespaced - logs of the namespaced Kubernetes objects.
/objects/namespaced/<namespaceName>/core/pods
- pods logs from a specified Kubernetes namespace.
/objects/namespaced/<namespaceName>/core/pods/<containerName>.prev.log
- logs of the pods from a specified Kubernetes namespace
that were previously removed or failed.
/objects/namespaced/<namespaceName>/core/pods/<ironicPodName>/syslog.log
- Technology Preview. Ironic pod logs of the bare metal clusters.
Note
Logs collected by the syslog container during the bootstrap phase
are not transferred to the management cluster during pivoting.
These logs are located in
/volume/log/ironic/ansible_conductor.log inside the Ironic
pod.
Each log entry of the management cluster logs contains a request ID that
identifies chronology of actions performed on a cluster or machine.
The format of the log entry is as follows:
<process ID>.[<subprocess ID>...<subprocess ID N>].req:<requestID>: <logMessage>
For example, os.machine.req:28 contains information about the task 28
applied to an OpenStack machine.
Since Container Cloud 2.22.0, the logging format has the following extended
structure for the admission-controller, storage-discovery, and all
supported <providerName>-provider services of a management cluster:
Informational level. Possible values: debug, info, warn,
error, panic.
ts
Time stamp in the <YYYY-MM-DDTHH:mm:ssZ> format. For example:
2022-11-14T21:37:23Z.
logger
Details on the process ID being logged:
<processID>
Primary process identifier. The list of possible values includes
bm, os, vsphere, iam, and license. The iam and
license values are available since Container Cloud 2.23.0.
<subProcessID(s)>
One or more secondary process identifiers. The list of possible values
includes cluster, machine, and controller. The controller
value is available since Container Cloud 2.23.0.
req
Request ID number that increases when a service performs the following
actions:
Receives a request from Kubernetes about creating, updating,
or deleting an object
Receives an HTTP request
Runs a background process
The request ID allows combining all operations performed with an object
within one request. For example, the result of a Machine object
creation, update of its statuses, and so on has the same request ID.
caller
Code line used to apply the corresponding action to an object.
msg
Description of a deployment or update phase. If empty, it contains an
error message with a stacktrace.
Depending on the type of issue found in logs, apply the corresponding fixes.
For example, if you detect the LoadBalancerERRORstate errors
during the bootstrap of an OpenStack-based management cluster,
contact your system administrator to fix the issue.
To troubleshoot other issues, refer to the corresponding section
in Troubleshooting.
The issue may occur because the default Docker network address
172.17.0.0/16 and/or the kind Docker network, which is used by
kind, overlap with your cloud address or other addresses
of the network configuration.
Workaround:
Log in to your local machine.
Verify routing to the IP addresses of the target cloud endpoints:
Obtain the IP address of your target cloud. For example:
If the routing is incorrect, change the IP address
of the default Docker bridge:
Create or edit /etc/docker/daemon.json by adding the "bip"
option:
{"bip":"192.168.91.1/24"}
Restart the Docker daemon:
sudosystemctlrestartdocker
If required, customize addresses for your kind Docker network
or any other additional Docker networks:
Remove the kind network:
dockernetworkrm'kind'
Choose from the following options:
Configure /etc/docker/daemon.json:
Note
The following steps are applied to to customize addresses
for the kind Docker network. Use these steps as an
example for any other additional Docker networks.
Add the following section to /etc/docker/daemon.json:
Docker pruning removes the user defined networks,
including 'kind'. Therefore,
every time after running the Docker pruning commands,
re-create the 'kind' network again
using the command above.
This section provides solutions to the issues that may occur while deploying
an OpenStack-based management cluster. To troubleshoot a managed cluster, see
Operations Guide: Troubleshooting.
If you execute the bootstrap.sh script from an OpenStack VM
that is running on the OpenStack environment used for bootstrapping
the management cluster, the following error messages may occur
that can be related to the MTU settings discrepancy:
If the MTU output values differ for docker0 and ens3, proceed
with the workaround below. Otherwise, inspect the logs further
to identify the root cause of the error messages.
Workaround:
In your OpenStack environment used for Mirantis Container
Cloud, log in to any machine with CLI access to OpenStack.
For example, you can create a new Ubuntu VM (separate from the bootstrap VM)
and install the python-openstackclient package on it.
Change the vXLAN MTU size for the VM to the required value
depending on your network infrastructure and considering your
physical network configuration, such as Jumbo frames, and so on.
This section provides solutions to the issues that may occur while deploying
a vSphere-based management cluster. To troubleshoot a managed cluster, see
Operations Guide: Troubleshooting.
Issues with virtual machines obtaining an IP may occur during the machines
deployment of the vSphere-based Container Cloud management or managed cluster
with IPAM enabled.
The issue symptoms are as follows:
On a cluster network with a DHCP server, the machine obtains a wrong
IP address that is most likely provided by the DHCP server.
The cluster deployment proceeds with unexpected IP addresses
that are not in the IPAM range.
On a cluster network without a DHCP server, the machine does not obtain
an IP address. The deployment freezes and fails by timeout.
To apply the issue resolution:
Verify that the cloud-init package version in the VM template is
19.4 or later. Older versions are affected by the
cloud-init bug.
cloud-init--version
Verify that the open-vm-tools package version is 11.0.5 or later.
vmtoolsd--version
vmware-toolbox-cmd--version
Verify that the /etc/cloud/cloud.cfg.d/99-DataSourceVMwareGuestInfo.cfg
file is present on the cluster and it is not empty.
Verify that the DataSourceVMwareGuestInfo.py file is present in the
cloud-init sources folder and is not empty.
To obtain the cloud-init folder:
python-c'import os; from cloudinit import sources; print(os.path.dirname(sources.__file__));'
If your deployment meets the requirements described in the verification steps
above but the issue still persists, rebuild the VM template as described in
Prepare the virtual machine template or contact Mirantis support.
This section describes how to configure authentication for Mirantis
Container Cloud depending on the external identity provider type
integrated to your deployment.
If you integrate LDAP for IAM to Mirantis Container Cloud,
add the required LDAP configuration to cluster.yaml.template
during the bootstrap of the management cluster.
Note
The example below defines the recommended non-anonymous
authentication type. If you require anonymous authentication,
replace the following parameters with authType: "none":
authType:"simple"bindCredential:""bindDn:""
To configure LDAP for IAM:
Open cluster.yaml.template stored in the following locations depending
on the cloud provider type:
Bare metal: templates/bm/cluster.yaml.template
OpenStack: templates/cluster.yaml.template
vSphere: templates/vsphere/cluster.yaml.template
Configure the keycloak:userFederation:providers:
and keycloak:userFederation:mappers: sections as required:
Verify that the userFederation section is located
on the same level as the initUsers section.
Verify that all attributes set in the mappers section
are defined for users in the specified LDAP system.
Missing attributes may cause authorization issues.
Now, return to the bootstrap instruction depending on the provider type
of your management cluster.
The instruction below applies to the DNS-based management
clusters. If you bootstrap a non-DNS-based management cluster,
configure Google OAuth IdP for Keycloak after bootstrap using the
official Keycloak documentation.
If you integrate Google OAuth external identity provider for IAM to
Mirantis Container Cloud, create the authorization credentials for IAM
in your Google OAuth account and configure cluster.yaml.template
during the bootstrap of the management cluster.
In the APIs Credentials menu, select
OAuth client ID.
In the window that opens:
In the Application type menu, select
Web application.
In the Authorized redirect URIs field, type in
<keycloak-url>/auth/realms/iam/broker/google/endpoint,
where <keycloak-url> is the corresponding DNS address.
Press Enter to add the URI.
Click Create.
A page with your client ID and client secret opens. Save these
credentials for further usage.
Log in to the bootstrap node.
Open cluster.yaml.template stored in the following locations depending
on the cloud provider type:
Bare metal: templates/bm/cluster.yaml.template
OpenStack: templates/cluster.yaml.template
vSphere: templates/vsphere/cluster.yaml.template
In the keycloak:externalIdP: section, add the following snippet
with your credentials created in previous steps:
The Mirantis Container Cloud APIs are implemented using the Kubernetes
CustomResourceDefinitions (CRDs) that enable you to expand the Kubernetes API.
For details, see API Reference.
You can operate Container Cloud using the kubectl
command-line tool that is based on the Kubernetes API.
For the kubectl reference, see the official
Kubernetes documentation.
The Container Cloud Operations Guide mostly contains manuals that describe
the Container Cloud web UI that is intuitive and easy to get started with.
Some sections are divided into a web UI instruction and an analogous
but more advanced CLI one.
Certain Container Cloud operations can be performed only using CLI
with the corresponding steps described in dedicated sections.
For details, refer to the required component section of this guide.
This tutorial applies only to the Container Cloud web UI users
with the m:kaas:namespace@operator or m:kaas:namespace@writer
access role assigned by the Infrastructure Operator.
To add a bare metal host, the m:kaas@operator or
m:kaas:namespace@bm-pool-operator role is required.
After you deploy the Mirantis Container Cloud management cluster,
you can start creating managed clusters that will be based on the same cloud
provider type that you have for the management or regional cluster:
OpenStack, bare metal, or vSphere.
The deployment procedure is performed using the Container Cloud web UI
and comprises the following steps:
Create a dedicated non-default project for managed clusters.
For a baremetal-based managed cluster, create and configure bare metal
hosts with corresponding labels for machines such as worker,
manager, or storage.
Create an initial cluster configuration depending on the provider type.
Add the required amount of machines with the corresponding configuration
to the managed cluster.
For a baremetal-based managed cluster, add a Ceph cluster.
Note
The Container Cloud web UI communicates with Keycloak
to authenticate users. Keycloak is exposed using HTTPS with
self-signed TLS certificates that are not trusted by web browsers.
The procedure below applies only to the Container Cloud web UI
users with the m:kaas@global-admin or m:kaas@writer access role
assigned by the infrastructure Operator.
The default project (Kubernetes namespace) in Container Cloud is dedicated
for management and regional clusters only. Managed clusters require a
separate project. You can create as many projects as required by your company
infrastructure.
To create a project for managed clusters using the Container Cloud web UI:
Log in to the Container Cloud web UI as m:kaas@global-admin or
m:kaas@writer.
In the Projects tab, click Create.
Type the new project name.
Click Create.
Generate a kubeconfig for a managed cluster using API¶
This section describes how to generate a managed cluster kubeconfig using
the Container Cloud API. You can also download a managed cluster kubeconfig
using the Download Kubeconfig option in the Container Cloud web
UI. For details, see Connect to a Mirantis Container Cloud cluster.
To generate a managed cluster kubeconfig using API:
The kubeconfig of your <username> that you can download through
the Container Cloud web UI using Download Kubeconfig located
under your <username> on the top-left of the page.
Obtain the <cluster> object of the <cluster_name> managed cluster:
Generate the managed cluster kubeconfig using the data from
<cluster.status> and <token> obtained in the previous steps.
Use the following template as an example:
Create and operate a baremetal-based managed cluster¶
After bootstrapping your baremetal-based Mirantis Container Cloud
management cluster as described in Deploy a baremetal-based management cluster,
you can start creating the baremetal-based managed clusters.
Before creating a bare metal managed cluster, add the required number
of bare metal hosts either using the Container Cloud web UI for a default
configuration or using CLI for an advanced configuration.
Specify the name of the user for accessing the BMC (IPMI user).
Password
Specify the password of the user for accessing the BMC (IPMI
password).
Boot MAC address
Specify the MAC address of the PXE network interface.
IP Address
Specify the IP address to access the BMC.
Label
Assign the machine label to the new host that defines which type of
machine may be deployed on this bare metal host. Only one label can
be assigned to a host. The supported labels include:
Manager
This label is selected and set by default.
Assign this label to the bare metal hosts that can be used
to deploy machines with the manager type. These hosts
must match the CPU and RAM requirements described in
Reference hardware configuration.
Worker
The host with this label may be used to deploy
the worker machine type. Assign this label to the bare metal hosts
that have sufficient CPU and RAM resources, as described in
Reference hardware configuration.
Storage
Assign this label to the bare metal hosts that have sufficient
storage devices to match Reference hardware configuration.
Hosts with this label will be used to deploy machines
with the storage type that run Ceph OSDs.
Click Create.
While adding the bare metal host, Container Cloud discovers and inspects
the hardware of the bare metal host and adds it to BareMetalHost.status
for future references.
During provisioning, baremetal-operator inspects the bare metal host
and moves it to the Preparing state. The host becomes ready to be linked
to a bare metal machine.
Verify the results of the hardware inspection to avoid unexpected errors
during the host usage:
In the BM Hosts tab, verify that the bare metal host
is registered and switched to one of the following statuses:
Preparing for a newly added host
Ready for a previously used host or for a host that is
already linked to a machine
Click the name of the newly added bare metal host.
In the window with the host details, scroll down to the
Hardware section.
Review the section and make sure that the number and models
of disks, network interface cards, and CPUs match the hardware
specification of the server.
If the hardware details are consistent with the physical server
specifications for all your hosts, proceed to
Add a managed baremetal cluster.
If you find any discrepancies in the hardware inspection results,
it might indicate that the server has hardware issues or
is not compatible with Container Cloud.
Log in to the host where your management cluster kubeconfig is located
and where kubectl is installed.
Select from the following options:
Since Container Cloud 2.21.0 and 2.21.1 for MOSK 22.5,
create a YAML file that describes the unique credentials of the
new bare metal host as a BareMetalHostCredential object.
In the metadata section, add a unique credentials name and the
name of the non-default project (namespace) dedicated for the
managed cluster being created.
In the spec section, add the IPMI user name and password in plain
text to access the Baseboard Management Controller (BMC). The password
will not be stored in the BareMetalHostCredential object but will
be erased and saved in an underlying Secret object.
Caution
Each bare metal host must have a unique
BareMetalHostCredential.
Before Container Cloud 2.21.0 or MOSK 22.5, create a
secret YAML file that describes the unique credentials of the new bare
metal host.
In the data section, add the IPMI user name and password in the
base64 encoding to access the BMC. To obtain the base64-encoded
credentials, you can use the following command in your Linux console:
echo-n<username|password>|base64
Caution
Each bare metal host must have a unique Secret.
In the metadata section, add the unique name of credentials and
the name of the non-default project (namespace) dedicated for
the managed cluster being created. To create a project, refer to
Create a project for managed clusters.
Apply the created YAML file with credentials to your deployment:
During provisioning, baremetal-operator inspects the bare metal host
and moves it to the Preparing state. The host becomes ready to be linked
to a bare metal machine.
During provisioning, the status changes as follows:
registering
inspecting
preparing
After BareMetalHost switches to the preparing stage, the
inspecting phase finishes and you can verify hardware information
available in the object status. For example:
The bare metal host profile is a Kubernetes custom resource.
It allows the operator to define how the storage devices
and the operating system are provisioned and configured.
This section describes the bare metal host profile default settings
and configuration of custom profiles for managed clusters
using Mirantis Container Cloud API. This procedure also applies
to a management cluster with a few differences described in
Customize the default bare metal host profile.
Note
You can view the created profiles in the
BM Host Profiles tab of the Container Cloud web UI.
Note
Using BareMetalHostProfile, you can configure LVM or mdadm-based
software RAID support during a management or managed cluster
creation. For details, see Configure RAID support.
This feature is available as Technology Preview. Use such
configuration for testing and evaluation purposes only. For the
Technology Preview feature definition, refer to Technology Preview features.
The default host profile requires three storage devices in the following
strict order:
Boot device and operating system storage
This device contains boot data and operating system data. It
is partitioned using the GUID Partition Table (GPT) labels.
The root file system is an ext4 file system
created on top of an LVM logical volume.
For a detailed layout, refer to the table below.
Local volumes device
This device contains an ext4 file system with directories mounted
as persistent volumes to Kubernetes. These volumes are used by
the Mirantis Container Cloud services to store its data,
including monitoring and identity databases.
Ceph storage device
This device is used as a Ceph datastore or Ceph OSD on managed clusters.
It is used as a Ceph datastore or Ceph OSD.
Warning
All data will be wiped during cluster deployment on devices
defined directly or indirectly in the fileSystems list of
BareMetalHostProfile. For example:
A raw device partition with a file system on it
A device partition in a volume group with a logical volume that has a
file system on it
An mdadm RAID device with a file system on it
An LVM RAID device with a file system on it
The wipe field is always considered true for these devices.
The false value is ignored.
Therefore, to prevent data loss, move the necessary data from these file
systems to another server beforehand, if required.
The following table summarizes the default configuration of the host system
storage set up by the Container Cloud bare metal management.
Default configuration of the bare metal host storage¶
Device/partition
Name/Mount point
Recommended size, GB
Description
/dev/sda1
bios_grub
4 MiB
The mandatory GRUB boot partition required for non-UEFI systems.
/dev/sda2
UEFI -> /boot/efi
0.2 GiB
The boot partition required for the UEFI boot mode.
/dev/sda3
config-2
64 MiB
The mandatory partition for the cloud-init configuration.
Used during the first host boot for initial configuration.
/dev/sda4
lvm_root_part
100% of the remaining free space in the LVM volume group
The main LVM physical volume that is used to create the root file system.
/dev/sdb
lvm_lvp_part -> /mnt/local-volumes
100% of the remaining free space in the LVM volume group
The LVM physical volume that is used to create the file system
for LocalVolumeProvisioner.
/dev/sdc
-
100% of the remaining free space in the LVM volume group
Clean raw disk that is used for the Ceph storage back end on
managed clusters.
If required, you can customize the default host storage configuration.
For details, see Create a custom host profile.
In addition to the default BareMetalHostProfile object installed
with Mirantis Container Cloud, you can create custom profiles
for managed clusters using Container Cloud API.
Note
The procedure below also applies to the Container Cloud
management clusters.
Warning
All data will be wiped during cluster deployment on devices
defined directly or indirectly in the fileSystems list of
BareMetalHostProfile. For example:
A raw device partition with a file system on it
A device partition in a volume group with a logical volume that has a
file system on it
An mdadm RAID device with a file system on it
An LVM RAID device with a file system on it
The wipe field is always considered true for these devices.
The false value is ignored.
Therefore, to prevent data loss, move the necessary data from these file
systems to another server beforehand, if required.
To create a custom bare metal host profile:
Select from the following options:
For a management cluster, log in to the bare metal seed node that will be
used to bootstrap the management cluster.
For a managed cluster, log in to the local machine where you management
cluster kubeconfig is located and where kubectl is installed.
Note
The management cluster kubeconfig is created automatically
during the last stage of the management cluster bootstrap.
Select from the following options:
For a management cluster, open
templates/bm/baremetalhostprofiles.yaml.template for editing.
For a managed cluster, create a new bare metal host profile
under the templates/bm/ directory.
Edit the host profile using the example template below to meet
your hardware configuration requirements:
Example template of a bare metal host profile
apiVersion:metal3.io/v1alpha1kind:BareMetalHostProfilemetadata:name:<profileName>namespace:<ManagedClusterProjectName># Add the name of the non-default project for the managed cluster# being created.spec:devices:# From the HW node, obtain the first device, which size is at least 120Gib.-device:minSize:120Giwipe:truepartitions:-name:bios_grubpartflags:-bios_grubsize:4Miwipe:true-name:uefipartflags:-espsize:200Miwipe:true-name:config-2size:64Miwipe:true-name:lvm_root_partsize:0wipe:true# From the HW node, obtain the second device, which size is at least 120Gib.# If a device exists but does not fit the size,# the BareMetalHostProfile will not be applied to the node.-device:minSize:120Giwipe:true# From the HW node, obtain the disk device with the exact name.-device:byName:/dev/nvme0n1minSize:120Giwipe:truepartitions:-name:lvm_lvp_partsize:0wipe:true# Example of wiping a device w\o partitioning it.# Mandatory for the case when a disk is supposed to be used for Ceph back end.# later-device:byName:/dev/sdewipe:truefileSystems:-fileSystem:vfatpartition:config-2-fileSystem:vfatmountPoint:/boot/efipartition:uefi-fileSystem:ext4logicalVolume:rootmountPoint:/-fileSystem:ext4logicalVolume:lvpmountPoint:/mnt/local-volumes/logicalVolumes:-name:rootsize:0vg:lvm_root-name:lvpsize:0vg:lvm_lvppostDeployScript:|#!/bin/bash -execho $(date) 'post_deploy_script done' >> /root/post_deploy_donepreDeployScript:|#!/bin/bash -execho $(date) 'pre_deploy_script done' >> /root/pre_deploy_donevolumeGroups:-devices:-partition:lvm_root_partname:lvm_root-devices:-partition:lvm_lvp_partname:lvm_lvpgrubConfig:defaultGrubOptions:-GRUB_DISABLE_RECOVERY="true"-GRUB_PRELOAD_MODULES=lvm-GRUB_TIMEOUT=20kernelParameters:sysctl:kernel.panic:"900"kernel.dmesg_restrict:"1"kernel.core_uses_pid:"1"fs.file-max:"9223372036854775807"fs.aio-max-nr:"1048576"fs.inotify.max_user_instances:"4096"vm.max_map_count:"262144"
To use multiple devices for LVM volume, use the
example template extract below for reference.
Caution
The following template extract contains only sections relevant
to LVM configuration with multiple PVs.
Expand the main template described in the previous step
with the configuration below if required.
Optional. Technology Preview. Configure support of the Redundant Array of
Independent Disks (RAID) that allows, for example, installing a cluster
operating system on a RAID device, refer to Configure RAID support.
Add or edit the mandatory parameters in the new BareMetalHostProfile
object. For the parameters description, see
API: BareMetalHostProfile spec.
The BareMetalHostProfile API allows configuring a host to use the
huge pages feature of the Linux kernel on managed clusters.
Note
Huge pages is a mode of operation of the Linux kernel. With
huge pages enabled, the kernel allocates the RAM in bigger
chunks, or pages. This allows a KVM (kernel-based virtual machine)
and VMs running on it to use the host RAM more efficiently
and improves the performance of VMs.
To enable huge pages in a custom bare metal host profile for a managed
cluster:
Log in to the local machine where you management
cluster kubeconfig is located and where kubectl is installed.
Note
The management cluster kubeconfig is created automatically
during the last stage of the management cluster bootstrap.
Open for editing or create a new bare metal host profile
under the templates/bm/ directory.
Edit the grubConfig section of the host profile spec using
the example below to configure the kernel boot parameters and
enable huge pages:
The example configuration above will allocate N huge pages of 1 GB each
on the server boot. The last hugepagesz parameter value is default unless
default_hugepagesz is defined. For details about possible values, see
official Linux kernel documentation.
Add the bare metal host profile to your management cluster:
This feature is available as Technology Preview. Use such
configuration for testing and evaluation purposes only.
For the Technology Preview feature definition, refer to Technology Preview features.
You can configure support of the software-based Redundant Array of Independent
Disks (RAID) using BareMetalHosProfile to set up an LVM or mdadm-based RAID
level 1 (raid1). If required, you can further configure RAID in the same
profile, for example, to install a cluster operating system onto a RAID device.
Caution
RAID configuration on already provisioned bare metal machines
or on an existing cluster is not supported.
To start using any kind of RAIDs, reprovisioning of machines
with a new BaremetalHostProfile is required.
Mirantis supports the raid1 type of RAID devices both
for LVM and mdadm.
Mirantis supports the raid0 type for the mdadm RAID
to be on par with the LVM linear type.
Mirantis recommends having at least two physical disks
for the raid0 and raid1 devices to prevent unnecessary
complexity.
Mirantis supports the raid10 type for mdadm RAID on
MOSK clusters. At least four physical disks
are required for this type of RAID.
Only an even number of disks can be used for a raid1 or
raid10 device.
This feature is available as Technology Preview. Use such
configuration for testing and evaluation purposes only.
For the Technology Preview feature definition, refer to Technology Preview features.
Warning
The EFI system partition partflags: ['esp'] must be
a physical partition in the main partition table of the disk, not under
LVM or mdadm software RAID.
During configuration of your custom bare metal host profile,
you can create an LVM-based software RAID device raid1 by adding
type: raid1 to the logicalVolume spec in BaremetalHostProfile.
The logicalVolume spec of the raid1 type requires
at least two devices (partitions) in volumeGroup where you
build a logical volume. For an LVM of the linear type,
one device is enough.
Note
The LVM raid1 requires additional space to store the raid1
metadata on a volume group, roughly 4 MB for each partition.
Therefore, you cannot create a logical volume of exactly the same
size as the partitions it works on.
For example, if you have two partitions of 10 GiB, the corresponding
raid1 logical volume size will be less than 10 GiB. For that
reason, you can either set size: 0 to use all available space
on the volume group, or set a smaller size than the partition size.
For example, use size: 9.9Gi instead of size: 10Gi
for the logical volume.
The following example illustrates an extract of BaremetalHostProfile
with / on the LVM raid1.
All data will be wiped during cluster deployment on devices
defined directly or indirectly in the fileSystems list of
BareMetalHostProfile. For example:
A raw device partition with a file system on it
A device partition in a volume group with a logical volume that has a
file system on it
An mdadm RAID device with a file system on it
An LVM RAID device with a file system on it
The wipe field is always considered true for these devices.
The false value is ignored.
Therefore, to prevent data loss, move the necessary data from these file
systems to another server beforehand, if required.
This feature is available as Technology Preview. Use such
configuration for testing and evaluation purposes only.
For the Technology Preview feature definition, refer to Technology Preview features.
Warning
The EFI system partition partflags: ['esp'] must be
a physical partition in the main partition table of the disk, not under
LVM or mdadm software RAID.
During configuration of your custom bare metal host profile as described in
Create a custom bare metal host profile, you can create an mdadm-based software RAID
device raid1 by describing the mdadm devices under the softRaidDevices
field in BaremetalHostProfile. For example:
The only two required fields to describe RAID devices are name
and devices. The devices field must describe at least two partitions
to build an mdadm RAID on it. For the mdadm RAID parameters, see
API: BareMetalHostProfile spec.
Caution
The mdadm RAID devices cannot be created on top of
LVM devices, as well as LVM devices cannot be created on top
of mdadm devices.
The following example illustrates an extract of BaremetalHostProfile
with / on the mdadm raid1 and some data storage on raid0:
All data will be wiped during cluster deployment on devices
defined directly or indirectly in the fileSystems list of
BareMetalHostProfile. For example:
A raw device partition with a file system on it
A device partition in a volume group with a logical volume that has a
file system on it
An mdadm RAID device with a file system on it
An LVM RAID device with a file system on it
The wipe field is always considered true for these devices.
The false value is ignored.
Therefore, to prevent data loss, move the necessary data from these file
systems to another server beforehand, if required.
The EFI system partition partflags: ['esp'] must be
a physical partition in the main partition table of the disk, not under
LVM or mdadm software RAID.
You can deploy Mirantis OpenStack for Kubernetes (MOSK) on local
software-based Redundant Array of Independent Disks (RAID) devices to
withstand failure of one device at a time.
Using a custom bare metal host profile, you can configure and create
an mdadm-based software RAID device of type raid10 if you have
an even number of devices available on your servers. At least four
storage devices are required for such RAID device.
During configuration of your custom bare metal host profile as described in
Create a custom bare metal host profile, create an mdadm-based software RAID device
raid10 by describing the mdadm devices under the softRaidDevices
field. For example:
The following fields in softRaidDevices describe RAID devices:
name
Name of the RAID device to refer to throughout the baremetalhostprofile.
devices
List of physical devices or partitions used to build a software RAID device.
It must include at least four partitions or devices to build a raid10
device.
level
Type or level of RAID used to create device. Set to raid10 or raid1
to create a device of the corresponding type.
All data will be wiped during cluster deployment on devices
defined directly or indirectly in the fileSystems list of
BareMetalHostProfile. For example:
A raw device partition with a file system on it
A device partition in a volume group with a logical volume that has a
file system on it
An mdadm RAID device with a file system on it
An LVM RAID device with a file system on it
The wipe field is always considered true for these devices.
The false value is ignored.
Therefore, to prevent data loss, move the necessary data from these file
systems to another server beforehand, if required.
This section instructs you on how to configure and deploy a managed cluster
that is based on the baremetal-based management cluster.
By default, Mirantis Container Cloud configures a single
interface on the cluster nodes, leaving all other physical interfaces intact.
With L2 networking templates, you can create advanced host networking
configurations for your clusters. For example, you can create bond
interfaces on top of physical interfaces on the host or use multiple subnets
to separate different types of network traffic.
You can use several host-specific L2 templates per one cluster
to support different hardware configurations. For example, you can create
L2 templates with different number and layout of NICs to be applied
to the specific machines of one cluster.
When you create a baremetal-based project, the exemplary templates
with the ipam/PreInstalledL2Template label are copied to this project.
These templates are preinstalled during the management cluster bootstrap.
Using the L2 Templates section of the Clusters tab
in the Container Cloud web UI, you can view a list of preinstalled templates
and the ones that you manually create before a cluster deployment.
Follow the procedures described in the below subsections to configure initial
settings and advanced network objects for your managed clusters.
Caution
Modification of L2 templates in use is allowed with a mandatory
validation step from the Infrastructure Operator to prevent accidental
cluster failures due to unsafe changes. The list of risks posed by modifying
L2 templates includes:
Services running on hosts cannot reconfigure automatically to switch to
the new IP addresses and/or interfaces.
Connections between services are interrupted unexpectedly, which can cause
data loss.
Incorrect configurations on hosts can lead to irrevocable loss of
connectivity between services and unexpected cluster partition or
disassembly.
This section instructs you on how to create initial configuration of a managed
cluster that is based on the baremetal-based management cluster through the
Mirantis Container Cloud web UI.
To create a managed cluster on bare metal:
Log in to the Container Cloud web UI with the m:kaas:namespace@operator or
m:kaas:namespace@writer permissions.
Switch to the required non-default project using the
Switch Project action icon located on top of the main left-side
navigation panel.
If your proxy requires a trusted CA certificate, select the
CA Certificate check box and paste a CA certificate for a MITM
proxy to the corresponding field or upload a certificate using
Upload Certificate.
For MOSK-based deployments, the possibility to use a MITM
proxy with a CA certificate is available since MOSK 23.1.
For MOSK-based deployments, the feature
support is available since MOSK 22.5.
Provider
LB host IP
The IP address of the load balancer endpoint that will be used to
access the Kubernetes API of the new cluster. This IP address
must be on the Combined/PXE network.
LB address range
The range of IP addresses that can be assigned to load balancers
for Kubernetes Services by MetalLB.
Kubernetes
Services CIDR blocks
The Kubernetes Services CIDR blocks.
For example, 10.233.0.0/18.
Pods CIDR blocks
The Kubernetes pods CIDR blocks.
For example, 10.233.64.0/18.
Note
The network subnet size of Kubernetes pods influences the number of
nodes that can be deployed in the cluster.
The default subnet size /18 is enough to create a cluster with
up to 256 nodes. Each node uses the /26 address blocks
(64 addresses), at least one address block is allocated per node.
These addresses are used by the Kubernetes pods with
hostNetwork:false. The cluster size may be limited
further when some nodes use more than one address block.
Configure StackLight:
Section
Parameter name
Description
StackLight
Enable Monitoring
Selected by default. Deselect to skip
StackLight deployment. You can also enable, disable, or configure
StackLight parameters after deploying a managed cluster. For details,
see Change a cluster configuration or Configure StackLight.
The logging mechanism performance depends on the cluster log load. In
case of a high load, you may need to increase the default resource requests
and limits for fluentdLogs. For details, see
StackLight configuration parameters: Resource limits.
HA Mode
Select to enable StackLight monitoring in the HA mode. For the
differences between HA and non-HA modes, see Deployment architecture.
StackLight Default Logs Severity Level
Log severity (verbosity) level for all StackLight components.
The default value for this parameter is
Default component log level that respects original defaults
of each StackLight component.
For details about severity levels, see Log verbosity.
StackLight Component Logs Severity Level
The severity level of logs for a specific StackLight component that
overrides the value of the
StackLight Default Logs Severity Level parameter.
For details about severity levels, see Log verbosity.
Expand the drop-down menu for a specific component to display
its list of available log levels.
OpenSearch
Logstash Retention Time
Available if you select Enable Logging. Specifies the
logstash-* index retention time.
Events Retention Time
Available if you select Enable Logging. Specifies the
kubernetes_events-* index retention time.
Notifications Retention
Available if you select Enable Logging. Specifies the
notification-* index retention time and is used for Mirantis
OpenStack for Kubernetes.
Persistent Volume Claim Size
Available if you select Enable Logging. The OpenSearch
persistent volume claim size.
Collected Logs Severity Level
Available if you select Enable Logging.
The minimum severity of all Container Cloud components logs collected in
OpenSearch. For details about severity levels, see Logging.
Prometheus
Retention Time
The Prometheus database retention period.
Retention Size
The Prometheus database retention size.
Persistent Volume Claim Size
The Prometheus persistent volume claim size.
Enable Watchdog Alert
Select to enable the Watchdog alert that fires
as long as the entire alerting pipeline is functional.
Custom Alerts
Specify alerting rules for new custom alerts or upload a YAML file
in the following exemplary format:
Select to enable notifications about resolved StackLight alerts.
Require TLS
Select to enable transmitting emails through TLS.
Email alerts configuration for StackLight
Fill out the following email alerts parameters as required:
To - the email address to send notifications to.
From - the sender address.
SmartHost - the SMTP host through which the emails are sent.
Authentication username - the SMTP user name.
Authentication password - the SMTP password.
Authentication identity - the SMTP identity.
Authentication secret - the SMTP secret.
StackLight Slack Alerts
Enable Slack alerts
Select to enable the StackLight Slack alerts.
Send Resolved
Select to enable notifications about resolved StackLight alerts.
Slack alerts configuration for StackLight
Fill out the following Slack alerts parameters as required:
API URL - The Slack webhook URL.
Channel - The channel to send notifications to, for example,
#channel-for-alerts.
StackLight optional settings
Enable Reference Application
Available since Container Cloud 2.22.0. Enables Reference Application
that is a small microservice application that enables
workload monitoring on non-MOSK managed clusters.
To simplify operations with L2 templates, before you start creating
them, inspect the general workflow of a network interface name gathering
and processing.
Network interface naming workflow:
The Operator creates a baremetalHost object.
The baremetalHost object executes the introspection stage
and becomes ready.
The Operator collects information about NIC count, naming, and so on
for further changes in the mapping logic.
At this stage, the NICs order in the object may randomly change
during each introspection, but the NICs names are always the same.
For more details, see Predictable Network Interface Names.
For example:
# Example commands:# kubectl -n managed-ns get bmh baremetalhost1 -o custom-columns='NAME:.metadata.name,STATUS:.status.provisioning.state'# NAME STATE# baremetalhost1 ready# kubectl -n managed-ns get bmh baremetalhost1 -o yaml# Example output:apiVersion:metal3.io/v1alpha1kind:BareMetalHost...status:...nics:-ip:fe80::ec4:7aff:fe6a:fb1f%eno2mac:0c:c4:7a:6a:fb:1fmodel:0x8086 0x1521name:eno2pxe:false-ip:fe80::ec4:7aff:fe1e:a2fc%ens1f0mac:0c:c4:7a:1e:a2:fcmodel:0x8086 0x10fbname:ens1f0pxe:false-ip:fe80::ec4:7aff:fe1e:a2fd%ens1f1mac:0c:c4:7a:1e:a2:fdmodel:0x8086 0x10fbname:ens1f1pxe:false-ip:192.168.1.151# Temp. PXE network adressmac:0c:c4:7a:6a:fb:1emodel:0x8086 0x1521name:eno1pxe:true...
The Operator selects from the following options:
Create an l2template object with the ifMapping configuration.
For details, see Create L2 templates.
The baremetal-provider service links the Machine object
to the baremetalHost object.
The kaas-ipam and baremetal-provider services collect hardware
information from the baremetalHost object and use it to configure
host networking and services.
The kaas-ipam service:
Spawns the IpamHost object.
Renders the l2template object.
Spawns the ipaddr object.
Updates the IpamHost object status with all rendered
and linked information.
The baremetal-provider service collects the rendered networking
information from the IpamHost object
The baremetal-provider service proceeds with the IpamHost object
provisioning.
Before creating an L2 template, ensure that you have the required subnets
that can be used in the L2 template to allocate IP addresses for the
managed cluster nodes.
Where required, create a number of subnets for a particular project
using the Subnet CR. A subnet has three logical scopes:
global - CR uses the default namespace.
A subnet can be used for any cluster located in any project.
namespaced - CR uses the namespace that corresponds to a particular project
where managed clusters are located. A subnet can be used for any cluster
located in the same project.
cluster - CR uses the namespace where the referenced cluster is located.
A subnet is only accessible to the cluster that
L2Template.spec.clusterRef refers to. The Subnet objects
with the cluster scope will be created for every new cluster.
You can have subnets with the same name in different projects.
In this case, the subnet that has the same project
as the cluster will be used. One L2 template may often reference several
subnets, those subnets may have different scopes in this case.
The IP address objects (IPaddr CR) that are allocated from subnets
always have the same project as their corresponding IpamHost objects,
regardless of the subnet scope.
You can create subnets using either the Container Cloud web UI or CLI.
Any Subnet object may contain ipam/SVC-<serviceName> labels.
All IP addresses allocated from the Subnet object that has service labels
defined, will inherit those labels.
When a particular IpamHost uses IP addresses allocated from such labeled
Subnet objects, the ServiceMap field in IpamHost.Status will
contain information about which IPs and interfaces correspond to which service
labels (that have been set in the Subnet objects). Using ServiceMap,
you can understand what IPs a