Create a managed cluster¶
Warning
This section only applies to Container Cloud 2.27.2 (Cluster release 16.2.2) or earlier versions. Since Container Cloud 2.27.3 (Cluster release 16.2.3), support for vSphere-based clusters is suspended. For details, see Deprecation notes.
This section describes how to create a VMware vSphere-based managed cluster using the Mirantis Container Cloud web UI of the vSphere-based management cluster.
To create a vSphere-based managed cluster:
Available since Container Cloud 2.24.0. Optional. Technology Preview. Enable custom host names for cluster machines. When enabled, any machine host name in a particular region matches the related
Machine
object name. For example, instead of the defaultkaas-node-<UID>
, a machine host name will bemaster-0
. The custom naming format is more convenient and easier to operate with.For details, see Configure host names for cluster machines.
If you enabled this feature during management cluster bootstrap, skip this step, as the feature applies to any cluster type.
Log in to the Container Cloud web UI with the
m:kaas:namespace@operator
orm:kaas:namespace@writer
permissions.Switch to the required non-
default
project using the Switch Project action icon located on top of the main left-side navigation panel.To create a project, refer to Create a project for managed clusters.
Optional. In the SSH Keys tab, click Add SSH Key to upload the public SSH key(s) for VMs creation.
In the Credentials tab:
Click Add Credential to add your vSphere credentials. You can either upload your vSphere
vsphere.yaml
configuration file or fill in the fields manually.Verify that the new credentials status is Ready. If the status is Error, hover over the status to determine the reason of the issue.
Optional. Enable proxy access to the cluster.
In the Proxies tab, configure proxy:
Click Add Proxy.
In the Add New Proxy wizard, fill out the form with the following parameters:
¶ Parameter
Description
Proxy Name
Name of the proxy server to use during cluster creation.
Region Removed in 2.26.0 (16.1.0 and 17.1.0)
From the drop-down list, select the required region.
HTTP Proxy
Add the HTTP proxy server domain name in the following format:
http://proxy.example.com:port
- for anonymous accesshttp://user:password@proxy.example.com:port
- for restricted access
HTTPS Proxy
Add the HTTPS proxy server domain name in the same format as for HTTP Proxy.
No Proxy
Comma-separated list of IP addresses or domain names. Mandatory to add
host[:port]
of the vCenter server.For implementation details, see Proxy and cache support.
If your proxy requires a trusted CA certificate, select the CA Certificate check box and paste a CA certificate for a MITM proxy to the corresponding field or upload a certificate using Upload Certificate.
For the list of Mirantis resources and IP addresses to be accessible from the Container Cloud clusters, see Requirements for a VMware vSphere-based cluster.
In the RHEL Licenses tab, click Add RHEL License and fill out the form with the following parameters:
¶ Parameter
Description
RHEL License Name
RHEL license name
Username (User/Password Registration)
User name to access the RHEL license
Password (User/Password Registration)
Password to access the RHEL license
Organization ID (Activation Key)
Organization key to register a user by
Activation Key (Activation Key)
Activation key to use for user registration
RPM URL (Activation Key)
Optional. URL from which to download RPM packages using RPM Package Manager
Pool IDs
Optional. Specify the pool IDs for RHEL licenses for Virtual Datacenters. Otherwise, Subscription Manager will select a subscription from the list of available and appropriate for the machines.
In the Clusters tab, click Create Cluster and fill out the form with the following parameters as required:
Configure general provider settings and the Kubernetes parameters:
Section
Parameter
Description
General Settings
Provider
Select vSphere.
Provider Credential
From the drop-down list, select the vSphere credentials name that you have previously created.
Release Version
The Container Cloud version.
Caution
Due to the known issue 40747, the Cluster release 16.0.0, which is not supported since Container Cloud 2.25.1 for greenfield deployments, is still available in the drop-down menu for managed clusters.
Do not select this Cluster release to prevent deployment failures. Select the latest supported version instead.
The issue 40747 is addressed in Container Cloud 2.26.1.
Proxy
Optional. From the drop-down list, select the proxy server name that you have previously created.
SSH Keys
From the drop-down list, select the SSH key name(s) that you have previously added for the SSH access to VMs.
Container Registry
From the drop-down list, select the Docker registry name that you have previously added using the Container Registries tab. For details, see Define a custom CA certificate for a private Docker registry.
Kubernetes
Node CIDR
Kubernetes nodes CIDR block. For example,
10.10.10.0/24
.Services CIDR Blocks
Kubernetes Services CIDR block. For example,
10.233.0.0/18
.Pods CIDR Blocks
Kubernetes pods CIDR block. For example,
10.233.64.0/18
.Note
The network subnet size of Kubernetes pods influences the number of nodes that can be deployed in the cluster. The default subnet size
/18
is enough to create a cluster with up to 256 nodes. Each node uses the/26
address blocks (64 addresses), at least one address block is allocated per node. These addresses are used by the Kubernetes pods withhostNetwork: false
. The cluster size may be limited further when some nodes use more than one address block.Provider
LB Host IP
IP address of the load balancer endpoint that will be used to access the Kubernetes API of the new cluster.
LB Address Range
MetalLB range of IP addresses that can be assigned to load balancers for Kubernetes Services.
vSphere
Machine Folder Path
Full path to the folder that will store the cluster machines metadata. Use the drop-down list to select the required item.
Note
Every drop-down list item of the vSphere section represents a short name of a particular vSphere resource, without the datacenter path. The Network Path drop-down list items also represent specific network types. Start typing the item name in the drop-down list field to filter the results and select the required item.
Network Path
Full path to a network for cluster machines. Use the drop-down list to select the required item.
Resource Pool Path
Full path to a resource pool where VMs will be created. Use the drop-down list to select the required item.
Datastore For Cluster
Full path to a storage for VMs disks. Use the drop-down list to select the required item.
Datastore For Cloud Provider
Full path to a storage for Kubernetes volumes. Use the drop-down list to select the required item.
SCSI Controller Type
SCSI controller type for VMs. Leave
pvscsi
as default.Enable IPAM
Enables IPAM. Set to
true
if a vSphere network has no DHCP server. Also, provide the following additional parameters for a proper network setup on machines using embedded IP address management (IPAM):Network CIDR
CIDR of the provided vSphere network. For example,
10.20.0.0/16
.Network Gateway
Gateway of the provided vSphere network.
DNS Name Servers
List of nameservers for the provided vSphere network.
Include Ranges
IP range for the cluster machines. Specify the range of the provided CIDR. For example,
10.20.0.100-10.20.0.200
.Exclude Ranges
Optional. IP ranges to be excluded from being assigned to the cluster machines. The MetalLB range and the load balancer IP address should not intersect with the addresses for IPAM. For example,
10.20.0.150-10.20.0.170
.Optional General Settings
Enable Secure Overlay
Experimental, not recommended for production deployments. Removed in Cluster releases 16.0.0 and 14.1.0.
Enable WireGuard for traffic encryption on the Kubernetes workloads network.
WireGuard configuration
Ensure that the Calico MTU size is at least 60 bytes smaller than the interface MTU size of the workload network. IPv4 WireGuard uses a 60-byte header. For details, see Set the MTU size for Calico.
Enable WireGuard by selecting the Enable WireGuard check box.
Caution
Changing this parameter on a running cluster causes a downtime that can vary depending on the cluster size.
For more details about WireGuard, see Calico documentation: Encrypt in-cluster pod traffic.
Parallel Upgrade Of Worker Machines
Available since the Cluster release 14.1.0.
The maximum number of the worker nodes to update simultaneously. It serves as an upper limit on the number of machines that are drained at a given moment of time. Defaults to
1
.You can configure this option after deployment before the cluster update.
Parallel Preparation For Upgrade Of Worker Machines
Available since the Cluster release 14.1.0.
The maximum number of worker nodes being prepared at a given moment of time, which includes downloading of new artifacts. It serves as a limit for the network load that can occur when downloading the files to the nodes. Defaults to
50
.You can configure this option after deployment before the cluster update.
Configure StackLight:
Section
Parameter name
Description
StackLight
Enable Monitoring
Selected by default. Deselect to skip StackLight deployment. You can also enable, disable, or configure StackLight parameters after deploying a managed cluster. For details, see Change a cluster configuration or Configure StackLight.
Enable Logging
Select to deploy the StackLight logging stack.
For details about the logging components, see Deployment architecture.
Note
The logging mechanism performance depends on the cluster log load. In case of a high load, you may need to increase the default resource requests and limits for
fluentdLogs
. For details, see StackLight configuration parameters: Resource limits.HA Mode
Select to enable StackLight monitoring in the HA mode. For the differences between HA and non-HA modes, see Deployment architecture.
StackLight Default Logs Severity Level
Log severity (verbosity) level for all StackLight components. The default value for this parameter is Default component log level that respects original defaults of each StackLight component. For details about severity levels, see MOSK Operations Guide: StackLight configuration parameters - Log verbosity.
StackLight Component Logs Severity Level
The severity level of logs for a specific StackLight component that overrides the value of the StackLight Default Logs Severity Level parameter. For details about severity levels, see MOSK Operations Guide: StackLight configuration parameters - Log verbosity.
Expand the drop-down menu for a specific component to display its list of available log levels.
OpenSearch
Logstash Retention Time
Skip this parameter since Container Cloud 2.26.0 (17.1.0, 16.1.0). It was removed from the code base and will be removed from the web UI in one of the following releases.
Available if you select Enable Logging. Specifies the
logstash-*
index retention time.Events Retention Time
Available if you select Enable Logging. Specifies the
kubernetes_events-*
index retention time.Notifications Retention
Available if you select Enable Logging. Specifies the
notification-*
index retention time and is used for Mirantis OpenStack for Kubernetes.Persistent Volume Claim Size
Available if you select Enable Logging. The OpenSearch persistent volume claim size.
Collected Logs Severity Level
Available if you select Enable Logging. The minimum severity of all Container Cloud components logs collected in OpenSearch. For details about severity levels, see MOSK Operations Guide: StackLight configuration parameters - Logging.
Prometheus
Retention Time
The Prometheus database retention period.
Retention Size
The Prometheus database retention size.
Persistent Volume Claim Size
The Prometheus persistent volume claim size.
Enable Watchdog Alert
Select to enable the Watchdog alert that fires as long as the entire alerting pipeline is functional.
Custom Alerts
Specify alerting rules for new custom alerts or upload a YAML file in the following exemplary format:
- alert: HighErrorRate expr: job:request_latency_seconds:mean5m{job="myjob"} > 0.5 for: 10m labels: severity: page annotations: summary: High request latency
For details, see Official Prometheus documentation: Alerting rules. For the list of the predefined StackLight alerts, see Operations Guide: Available StackLight alerts.
StackLight Email Alerts
Enable Email Alerts
Select to enable the StackLight email alerts.
Send Resolved
Select to enable notifications about resolved StackLight alerts.
Require TLS
Select to enable transmitting emails through TLS.
Email alerts configuration for StackLight
Fill out the following email alerts parameters as required:
To - the email address to send notifications to.
From - the sender address.
SmartHost - the SMTP host through which the emails are sent.
Authentication username - the SMTP user name.
Authentication password - the SMTP password.
Authentication identity - the SMTP identity.
Authentication secret - the SMTP secret.
StackLight Slack Alerts
Enable Slack alerts
Select to enable the StackLight Slack alerts.
Send Resolved
Select to enable notifications about resolved StackLight alerts.
Slack alerts configuration for StackLight
Fill out the following Slack alerts parameters as required:
API URL - The Slack webhook URL.
Channel - The channel to send notifications to, for example, #channel-for-alerts.
StackLight optional settings
Enable Reference Application
Unsupported since Container Cloud 2.28.3 (16.3.3). Available since 2.22.0 (11.6.0). Enables Reference Application that is a small microservice application that enables workload monitoring on non-MOSK managed clusters.
Note
For the feature support on MOSK deployments, refer to MOSK documentation: Deploy your first cloud application using automation.
Disabled by default. You can also enable this option after deployment from the Configure cluster menu.
Available since Container Cloud 2.24.0 and 2.24.2 for MOSK 23.2. Optional. Technology Preview. Enable the Linux Audit daemon auditd to monitor activity of cluster processes and prevent potential malicious activity.
Configuration for auditd
In the
Cluster
object, add the auditd parameters:spec: providerSpec: value: audit: auditd: enabled: <bool> enabledAtBoot: <bool> backlogLimit: <int> maxLogFile: <int> maxLogFileAction: <string> maxLogFileKeep: <int> mayHaltSystem: <bool> presetRules: <string> customRules: <string> customRulesX32: <text> customRulesX64: <text>
Configuration parameters for auditd:
enabled
Boolean, default -
false
. Enables theauditd
role to install the auditd packages and configure rules. CIS rules: 4.1.1.1, 4.1.1.2.enabledAtBoot
Boolean, default -
false
. Configures grub to audit processes that can be audited even if they start up prior to auditd startup. CIS rule: 4.1.1.3.backlogLimit
Integer, default - none. Configures the backlog to hold records. If during boot
audit=1
is configured, the backlog holds 64 records. If more than 64 records are created during boot, auditd records will be lost with a potential malicious activity being undetected. CIS rule: 4.1.1.4.maxLogFile
Integer, default - none. Configures the maximum size of the audit log file. Once the log reaches the maximum size, it is rotated and a new log file is created. CIS rule: 4.1.2.1.
maxLogFileAction
String, default - none. Defines handling of the audit log file reaching the maximum file size. Allowed values:
keep_logs
- rotate logs but never delete themrotate
- add a cron job to compress rotated log files and keep maximum 5 compressed files.compress
- compress log files and keep them under the/var/log/auditd/
directory. Requiresauditd_max_log_file_keep
to be enabled.
CIS rule: 4.1.2.2.
maxLogFileKeep
Integer, default -
5
. Defines the number of compressed log files to keep under the/var/log/auditd/
directory. Requiresauditd_max_log_file_action=compress
. CIS rules - none.mayHaltSystem
Boolean, default -
false
. Halts the system when the audit logs are full. Applies the following configuration:space_left_action = email
action_mail_acct = root
admin_space_left_action = halt
CIS rule: 4.1.2.3.
customRules
String, default - none. Base64-encoded content of the
60-custom.rules
file for any architecture. CIS rules - none.customRulesX32
String, default - none. Base64-encoded content of the
60-custom.rules
file for thei386
architecture. CIS rules - none.customRulesX64
String, default - none. Base64-encoded content of the
60-custom.rules
file for thex86_64
architecture. CIS rules - none.presetRules
String, default - none. Comma-separated list of the following built-in preset rules:
access
actions
delete
docker
identity
immutable
logins
mac-policy
modules
mounts
perm-mod
privileged
scope
session
system-locale
time-change
Since Container Cloud 2.28.0 (Cluster releases 17.3.0 and 16.3.0) in the Technology Preview scope, you can collect some of the preset rules indicated above as groups and use them in
presetRules
:ubuntu-cis-rules
- this group contains rules to comply with the Ubuntu CIS Benchmark recommendations, including the following CIS Ubuntu 20.04 v2.0.1 rules:scope
- 5.2.3.1actions
- same as 5.2.3.2time-change
- 5.2.3.4system-locale
- 5.2.3.5privileged
- 5.2.3.6access
- 5.2.3.7identity
- 5.2.3.8
perm-mod
- 5.2.3.9mounts
- 5.2.3.10session
- 5.2.3.11logins
- 5.2.3.12delete
- 5.2.3.13mac-policy
- 5.2.3.14modules
- 5.2.3.19
docker-cis-rules
- this group contains rules to comply with Docker CIS Benchmark recommendations, including thedocker
Docker CIS v1.6.0 rules 1.1.3 - 1.1.18.
You can also use two additional keywords inside
presetRules
:none
- select no built-in rules.all
- select all built-in rules. When using this keyword, you can add the!
prefix to a rule name to exclude some rules. You can use the!
prefix for rules only if you add theall
keyword as the first rule. Place a rule with the!
prefix only after theall
keyword.
Example configurations:
presetRules: none
- disable all preset rulespresetRules: docker
- enable only thedocker
rulespresetRules: access,actions,logins
- enable only theaccess
,actions
, andlogins
rulespresetRules: ubuntu-cis-rules
- enable all rules from theubuntu-cis-rules
grouppresetRules: docker-cis-rules,actions
- enable all rules from thedocker-cis-rules
group and theactions
rulepresetRules: all
- enable all preset rulespresetRules: all,!immutable,!sessions
- enable all preset rules exceptimmutable
andsessions
CIS controls
4.1.3 (time-change
)4.1.4 (identity
)4.1.5 (system-locale
)4.1.6 (mac-policy
)4.1.7 (logins
)4.1.8 (session
)4.1.9 (perm-mod
)4.1.10 (access
)4.1.11 (privileged
)4.1.12 (mounts
)4.1.13 (delete
)4.1.14 (scope
)4.1.15 (actions
)4.1.16 (modules
)4.1.17 (immutable
)Docker CIS controls
1.1.41.1.81.1.101.1.121.1.131.1.151.1.161.1.171.1.181.2.31.2.41.2.51.2.61.2.71.2.101.2.11
Click Create.
To monitor the cluster readiness, hover over the status icon of a specific cluster in the Status column of the Clusters page.
Once the orange blinking status icon becomes green and Ready, the cluster deployment or update is complete.
You can monitor live deployment status of the following cluster components:
Component
Description
Bastion
For the OpenStack-based management clusters, the Bastion node IP address status that confirms the Bastion node creation
Helm
Installation or upgrade status of all Helm releases
Kubelet
Readiness of the node in a Kubernetes cluster, as reported by kubelet
Kubernetes
Readiness of all requested Kubernetes objects
Nodes
Equality of the requested nodes number in the cluster to the number of nodes having the
Ready
LCM statusOIDC
Readiness of the cluster OIDC configuration
StackLight
Health of all StackLight-related objects in a Kubernetes cluster
Swarm
Readiness of all nodes in a Docker Swarm cluster
LoadBalancer
Readiness of the Kubernetes API load balancer
ProviderInstance
Readiness of all machines in the underlying infrastructure (virtual or bare metal, depending on the provider type)
Graceful Reboot
Readiness of a cluster during a scheduled graceful reboot, available since Cluster releases 15.0.1 and 14.0.0.
Infrastructure Status
Available since Container Cloud 2.25.0 for bare metal and OpenStack providers. Readiness of the following cluster components:
Bare metal: the
MetalLBConfig
object along with MetalLB and DHCP subnets.OpenStack: cluster network, routers, load balancers, and Bastion along with their ports and floating IPs.
LCM Operation
Available since Container Cloud 2.26.0 (Cluster releases 17.1.0 and 16.1.0). Health of all LCM operations on the cluster and its machines.
LCM Agent
Available since Container Cloud 2.27.0 (Cluster releases 17.2.0 and 16.2.0). Health of all LCM agents on cluster machines and the status of LCM agents update to the version from the current Cluster release.
For the history of a cluster deployment or update, refer to Inspect the history of a cluster and machine deployment or update.
Available since Container Cloud 2.24.0 and 2.24.2 for MOSK 23.2. Optional. Technology Preview. Enable the Linux Audit daemon auditd to monitor activity of cluster processes and prevent potential malicious activity.
Configuration for auditd
In the
Cluster
object, add the auditd parameters:spec: providerSpec: value: audit: auditd: enabled: <bool> enabledAtBoot: <bool> backlogLimit: <int> maxLogFile: <int> maxLogFileAction: <string> maxLogFileKeep: <int> mayHaltSystem: <bool> presetRules: <string> customRules: <string> customRulesX32: <text> customRulesX64: <text>
Configuration parameters for auditd:
enabled
Boolean, default -
false
. Enables theauditd
role to install the auditd packages and configure rules. CIS rules: 4.1.1.1, 4.1.1.2.enabledAtBoot
Boolean, default -
false
. Configures grub to audit processes that can be audited even if they start up prior to auditd startup. CIS rule: 4.1.1.3.backlogLimit
Integer, default - none. Configures the backlog to hold records. If during boot
audit=1
is configured, the backlog holds 64 records. If more than 64 records are created during boot, auditd records will be lost with a potential malicious activity being undetected. CIS rule: 4.1.1.4.maxLogFile
Integer, default - none. Configures the maximum size of the audit log file. Once the log reaches the maximum size, it is rotated and a new log file is created. CIS rule: 4.1.2.1.
maxLogFileAction
String, default - none. Defines handling of the audit log file reaching the maximum file size. Allowed values:
keep_logs
- rotate logs but never delete themrotate
- add a cron job to compress rotated log files and keep maximum 5 compressed files.compress
- compress log files and keep them under the/var/log/auditd/
directory. Requiresauditd_max_log_file_keep
to be enabled.
CIS rule: 4.1.2.2.
maxLogFileKeep
Integer, default -
5
. Defines the number of compressed log files to keep under the/var/log/auditd/
directory. Requiresauditd_max_log_file_action=compress
. CIS rules - none.mayHaltSystem
Boolean, default -
false
. Halts the system when the audit logs are full. Applies the following configuration:space_left_action = email
action_mail_acct = root
admin_space_left_action = halt
CIS rule: 4.1.2.3.
customRules
String, default - none. Base64-encoded content of the
60-custom.rules
file for any architecture. CIS rules - none.customRulesX32
String, default - none. Base64-encoded content of the
60-custom.rules
file for thei386
architecture. CIS rules - none.customRulesX64
String, default - none. Base64-encoded content of the
60-custom.rules
file for thex86_64
architecture. CIS rules - none.presetRules
String, default - none. Comma-separated list of the following built-in preset rules:
access
actions
delete
docker
identity
immutable
logins
mac-policy
modules
mounts
perm-mod
privileged
scope
session
system-locale
time-change
Since Container Cloud 2.28.0 (Cluster releases 17.3.0 and 16.3.0) in the Technology Preview scope, you can collect some of the preset rules indicated above as groups and use them in
presetRules
:ubuntu-cis-rules
- this group contains rules to comply with the Ubuntu CIS Benchmark recommendations, including the following CIS Ubuntu 20.04 v2.0.1 rules:scope
- 5.2.3.1actions
- same as 5.2.3.2time-change
- 5.2.3.4system-locale
- 5.2.3.5privileged
- 5.2.3.6access
- 5.2.3.7identity
- 5.2.3.8
perm-mod
- 5.2.3.9mounts
- 5.2.3.10session
- 5.2.3.11logins
- 5.2.3.12delete
- 5.2.3.13mac-policy
- 5.2.3.14modules
- 5.2.3.19
docker-cis-rules
- this group contains rules to comply with Docker CIS Benchmark recommendations, including thedocker
Docker CIS v1.6.0 rules 1.1.3 - 1.1.18.
You can also use two additional keywords inside
presetRules
:none
- select no built-in rules.all
- select all built-in rules. When using this keyword, you can add the!
prefix to a rule name to exclude some rules. You can use the!
prefix for rules only if you add theall
keyword as the first rule. Place a rule with the!
prefix only after theall
keyword.
Example configurations:
presetRules: none
- disable all preset rulespresetRules: docker
- enable only thedocker
rulespresetRules: access,actions,logins
- enable only theaccess
,actions
, andlogins
rulespresetRules: ubuntu-cis-rules
- enable all rules from theubuntu-cis-rules
grouppresetRules: docker-cis-rules,actions
- enable all rules from thedocker-cis-rules
group and theactions
rulepresetRules: all
- enable all preset rulespresetRules: all,!immutable,!sessions
- enable all preset rules exceptimmutable
andsessions
CIS controls
4.1.3 (time-change
)4.1.4 (identity
)4.1.5 (system-locale
)4.1.6 (mac-policy
)4.1.7 (logins
)4.1.8 (session
)4.1.9 (perm-mod
)4.1.10 (access
)4.1.11 (privileged
)4.1.12 (mounts
)4.1.13 (delete
)4.1.14 (scope
)4.1.15 (actions
)4.1.16 (modules
)4.1.17 (immutable
)Docker CIS controls
1.1.41.1.81.1.101.1.121.1.131.1.151.1.161.1.171.1.181.2.31.2.41.2.51.2.61.2.71.2.101.2.11
Proceed with Add a machine.
See also