Mirantis Container Cloud Operations Guide¶

Create a managed cluster¶

This section instructs you on how to configure and deploy a managed cluster that is based on the baremetal-based management cluster through the Mirantis Container Cloud web UI.

To create a managed cluster on bare metal:

1. Recommended. Verify that you have successfully configured an L2 template for a new cluster as described in Advanced networking configuration. You may skip this step if you do not require L2 separation for network traffic.

2. Optional. Create a custom bare metal host profile depending on your needs as described in Create a custom bare metal host profile.

3. Log in to the Container Cloud web UI with the writer permissions.

4. Switch to the required project using the Switch Project action icon located on top of the main left-side navigation panel.

5. In the SSH keys tab, click Add SSH Key to upload the public SSH key that will be used for the SSH access to VMs.

6. In the Clusters tab, click Create Cluster.

7. Configure the new cluster in the Create New Cluster wizard that opens:

   1. Define general and Kubernetes parameters:

      Create new cluster: General, Provider, and Kubernetes¶

      Section	Parameter name	Description
      General settings	Cluster name	The cluster name.
      	Provider	Select Baremetal.
      	Region	From the drop-down list, select Baremetal.
      	Release version	The Container Cloud version.
      	SSH keys	From the drop-down list, select the SSH key name that you have previously added for SSH access to the bare metal hosts.
      Provider	LB host IP	The IP address of the load balancer endpoint that will be used to access the Kubernetes API of the new cluster. This IP address must be on the Combined/PXE network.
      	LB address range	The range of IP addresses that can be assigned to load balancers for Kubernetes Services by MetalLB.
      Kubernetes	Node CIDR	Not applicable to bare metal. Set to an example value 10.10.10.0/24.
      	Services CIDR blocks	The Kubernetes Services CIDR blocks. For example, 10.233.0.0/18.
      	Pods CIDR blocks	The Kubernetes pods CIDR blocks. For example, 10.233.64.0/18.

   2. Configure StackLight:

      StackLight configuration¶

      Section	Parameter name	Description
      StackLight	Enable Monitoring	Selected by default. Deselect to skip StackLight deployment. Note You can also enable, disable, or configure StackLight parameters after deploying a managed cluster. For details, see Change a cluster configuration or Configure StackLight.
      	Enable Logging	Select to deploy the StackLight logging stack. For details about the logging components, see Reference Architecture: StackLight deployment architecture.
      	HA Mode	Select to enable StackLight monitoring in the HA mode. For the differences between HA and non-HA modes, see Reference Architecture: StackLight deployment architecture.
      Elasticsearch	Retention Time	The Elasticsearch logs retention period in Logstash.
      	Persistent Volume Claim Size	The Elasticsearch persistent volume claim size.
      Prometheus	Retention Time	The Prometheus database retention period.
      	Retention Size	The Prometheus database retention size.
      	Persistent Volume Claim Size	The Prometheus persistent volume claim size.
      	Enable Watchdog Alert	Select to enable the Watchdog alert that fires as long as the entire alerting pipeline is functional.
      	Custom Alerts	Specify alerting rules for new custom alerts or upload a YAML file in the following exemplary format: - alert: HighErrorRate expr: job:request_latency_seconds:mean5m{job="myjob"} > 0.5 for: 10m labels: severity: page annotations: summary: High request latency For details, see Official Prometheus documentation: Alerting rules. For the list of the predefined StackLight alerts, see Operations Guide: Available StackLight alerts.
      StackLight Email Alerts	Enable Email Alerts	Select to enable the StackLight email alerts.
      	Send Resolved	Select to enable notifications about resolved StackLight alerts.
      	Require TLS	Select to enable transmitting emails through TLS.
      	Email alerts configuration for StackLight	Fill out the following email alerts parameters as required: To - the email address to send notifications to. From - the sender address. SmartHost - the SMTP host through which the emails are sent. Authentication username - the SMTP user name. Authentication password - the SMTP password. Authentication identity - the SMTP identity. Authentication secret - the SMTP secret.
      StackLight Slack Alerts	Enable Slack alerts	Select to enable the StackLight Slack alerts.
      	Send Resolved	Select to enable notifications about resolved StackLight alerts.
      	Slack alerts configuration for StackLight	Fill out the following Slack alerts parameters as required: API URL - The Slack webhook URL. Channel - The channel to send notifications to, for example, #channel-for-alerts.

8. Click Create.

   To view the deployment status, verify the cluster status on the Clusters page. Once the orange blinking dot near the cluster name disappears, the deployment is complete.

Now, proceed to Add a bare metal host.

Add a bare metal host¶

This section describes how to add a bare metal host to a newly created managed cluster using either the Container Cloud web UI or CLI for an advanced configuration.

Add a machine¶

This section describes how to add a machine to a newly created managed cluster using either the Mirantis Container Cloud web UI or CLI for an advanced configuration.

apiVersion: cluster.k8s.io/v1alpha1
kind: Machine
metadata:
  finalizers:
  - foregroundDeletion
  - machine.cluster.sigs.k8s.io
  labels:
    cluster.sigs.k8s.io/cluster-name: kaas-mgmt
    cluster.sigs.k8s.io/control-plane: "true"
    kaas.mirantis.com/provider: baremetal
    kaas.mirantis.com/region: region-one
spec:
  providerSpec:
    value:
      apiVersion: baremetal.k8s.io/v1alpha1
      hostSelector:
        matchLabels:
          baremetal: hw-master-0
      image: {}
      kind: BareMetalMachineProviderSpec
      l2TemplateIfMappingOverride:
      - eno1
      - eno0
      - enp0s1
      - enp0s2

Add a Ceph cluster¶

After you add machines to your new bare metal managed cluster as described in Add a machine, you can create a Ceph cluster on top of this managed cluster using the Mirantis Container Cloud web UI.

For an advanced configuration through the KaaSCephCluster CR, see Ceph advanced configuration.

The procedure below enables you to create a Ceph cluster with minimum three Ceph nodes that provides persistent volumes to the Kubernetes workloads in the managed cluster.

To create a Ceph cluster in the managed cluster:

1. Log in to the Container Cloud web UI with the writer permissions.

2. Switch to the required project using the Switch Project action icon located on top of the main left-side navigation panel.

3. In the Clusters tab, click the required cluster name. The Cluster page with the Machines and Ceph clusters lists opens.

4. In the Ceph Clusters block, click Create Cluster.

5. Configure the Ceph cluster in the Create New Ceph Cluster wizard that opens:

   Create new Ceph cluster¶

   Section	Parameter name	Description
   General settings	Name	The Ceph cluster name.
   	Cluster Network	Replication network for Ceph OSDs. Must contain the CIDR definition and match the corresponding values of the cluster L2Template object or the environment network values.
   	Public Network	Public network for Ceph data. Must contain the CIDR definition and match the corresponding values of the cluster L2Template object or the environment network values.
   	Enable OSDs LCM	Select to enable LCM for Ceph OSDs.
   Machines / Machine #1-3	Select machine	Select the name of the Kubernetes machine that will host the corresponding Ceph node in the Ceph cluster.
   	Manager, Monitor	Select the required Ceph services to install on the Ceph node.
   	Devices	Select the disk that Ceph will use. Warning Do not select the device for system services, for example, sda.
   	Enable Object Storage Available since 2.5.0	Select to enable the single-instance RGW Object Storage.

6. To add more Ceph nodes to the new Ceph cluster, click + next to any Ceph Machine title in the Machines tab. Configure a Ceph node as required.

   Warning

   Do not add more than 3 Manager and/or Monitor services to the Ceph cluster.

7. After you add and configure all nodes in your Ceph cluster, click Create.

Once done, verify your Ceph cluster as described in Verify Ceph.

Delete a managed cluster¶

Due to a development limitation in baremetal operator, deletion of a managed cluster requires preliminary deletion of the worker machines running on the cluster.

Using the Container Cloud web UI, first delete worker machines one by one until you hit the minimum of 2 workers for an operational cluster. After that, you can delete the cluster with the remaining workers and managers.

To delete a baremetal-based managed cluster:

1. Log in to the Mirantis Container Cloud web UI with the writer permissions.

2. Switch to the required project using the Switch Project action icon located on top of the main left-side navigation panel.

3. In the Clusters tab, click the required cluster name to open the list of machines running on it.

4. Click the More action icon in the last column of the worker machine you want to delete and select Delete. Confirm the deletion.

5. Repeat the step above until you have 2 workers left.

6. In the Clusters tab, click the More action icon in the last column of the required cluster and select Delete.

7. Verify the list of machines to be removed. Confirm the deletion.

8. Optional. If you do not plan to reuse the credentials of the deleted cluster, delete them:

   1. In the Credentials tab, click the Delete credential action icon next to the name of the credentials to be deleted.

   2. Confirm the deletion.

   Warning

   You can delete credentials only after deleting the managed cluster they relate to.

Deleting a cluster automatically frees up the resources allocated for this cluster, for example, instances, load balancers, networks, floating IPs, and so on.

Advanced networking configuration¶

By default, Mirantis Container Cloud configures a single interface on the cluster nodes, leaving all other physical interfaces intact.

With L2 networking templates, you can create advanced host networking configurations for your clusters. For example, you can create bond interfaces on top of physical interfaces on the host or use multiple subnets to separate different types of network traffic.

You can use several host-specific L2 templates per one cluster to support different hardware configurations. For example, you can create L2 templates with different number and layout of NICs to be applied to the specific machines of one cluster.

When you create a baremetal-based project, the exemplary templates with the ipam/PreInstalledL2Template label are copied to this project. These templates are preinstalled during the management cluster bootstrap.

Follow the procedures below to create L2 templates for your managed clusters.

apiVersion: ipam.mirantis.com/v1alpha1
kind: L2Template
metadata:
  name: test-managed
  namespace: managed-ns
spec:
  clusterRef: managed-cluster
  autoIfMappingPrio:
    - provision
    - eno
    - ens
    - enp
  npTemplate: |
    version: 2
    ethernets:
      ten10gbe0s0:
        dhcp4: false
        dhcp6: false
        match:
          macaddress: {{mac 2}}
        set-name: {{nic 2}}
      ten10gbe0s1:
        dhcp4: false
        dhcp6: false
        match:
          macaddress: {{mac 3}}
        set-name: {{nic 3}}
    bonds:
      bond0:
        interfaces:
          - ten10gbe0s0
          - ten10gbe0s1
    bridges:
      k8s-ext:
        interfaces: [bond0]
        addresses:
          - {{ip "k8s-ext:demo-ext"}}
      k8s-pods:
        interfaces: [bond0]
        addresses:
          - {{ip "k8s-pods:demo-pods"}}

apiVersion: ipam.mirantis.com/v1alpha1
kind: L2Template
metadata:
  name: test-managed
  namespace: managed-ns
spec:
  clusterRef: managed-cluster
  autoIfMappingPrio:
    - provision
    - eno
    - ens
    - enp
  l3Layout:
    - subnetName: pxe-subnet
      scope:      global
    - subnetName: subnet-1
      subnetPool: kaas-mgmt
      scope:      namespace
    - subnetName: subnet-2
      subnetPool: kaas-mgmt
      scope:      cluster
  npTemplate: |
    version: 2
    ethernets:
      onboard1gbe0:
        dhcp4: false
        dhcp6: false
        match:
          macaddress: {{mac 0}}
        set-name: {{nic 0}}
        # IMPORTANT: do not assign an IP address here explicitly
        # to prevent IP duplication issues. The IP will be assigned
        # automatically by the bootstrapping engine.
        # addresses: []
      onboard1gbe1:
        dhcp4: false
        dhcp6: false
        match:
          macaddress: {{mac 1}}
        set-name: {{nic 1}}
      ten10gbe0s0:
        dhcp4: false
        dhcp6: false
        match:
          macaddress: {{mac 2}}
        set-name: {{nic 2}}
        addresses:
          - {{ip "2:subnet-1"}}
      ten10gbe0s1:
        dhcp4: false
        dhcp6: false
        match:
          macaddress: {{mac 3}}
        set-name: {{nic 3}}
        addresses:
          - {{ip "3:subnet-2"}}

Create a custom bare metal host profile¶

The bare metal host profile is a Kubernetes custom resource. It allows the operator to define how the storage devices and the operating system are provisioned and configured.

This section describes the bare metal host profile default settings and configuration of custom profiles for managed clusters using Mirantis Container Cloud API. This procedure also applies to a management cluster with a few differences described in Deployment Guide: Customize the default bare metal host profile.

Create a managed cluster¶

This section describes how to create an OpenStack-based managed cluster using the Mirantis Container Cloud web UI of the OpenStack-based management cluster.

To create an OpenStack-based managed cluster:

1. Log in to the Container Cloud web UI with the writer permissions.

2. Switch to the required project using the Switch Project action icon located on top of the main left-side navigation panel.

3. In the SSH Keys tab, click Add SSH Key to upload the public SSH key that will be used for the OpenStack VMs creation.

4. In the Credentials tab:

   1. Click Add Credential to add your OpenStack credentials. You can either upload your OpenStack clouds.yaml configuration file or fill in the fields manually.

   2. Verify that the new credentials status is Ready. If the status is Error, hover over the status to determine the reason of the issue.

5. ^{Available since 2.5.0} Optional. In the Proxies tab, enable proxy access to the managed cluster:

   1. Click Add Proxy.

   2. In the Add New Proxy wizard, fill out the form with the following parameters:

      Proxy configuration¶

      Parameter	Description
      Proxy Name	Name of the proxy server to use during a managed cluster creation.
      Region	From the drop-down list, select the required region.
      HTTP Proxy	Add the HTTP proxy server domain name in the following format: http://proxy.example.com:port - for anonymous access http://user:password@proxy.example.com:port - for restricted access
      HTTPS Proxy	Add the HTTPS proxy server domain name in the same format as for HTTP Proxy.
      No Proxy	Comma-separated list of IP addresses or domain names.

      For the list of Mirantis resources and IP addresses to be accessible from the Container Cloud clusters, see Reference Architecture: Hardware and system requirements.

6. In the Clusters tab, click Create Cluster and fill out the form with the following parameters as required:

   1. Configure general settings and the Kubernetes parameters:

      Managed cluster configuration¶

      Section	Parameter	Description
      General Settings	Name	Cluster name
      	Provider	Select OpenStack
      	Provider Credential	From the drop-down list, select the OpenStack credentials name that you have previously created.
      	Release Version	The Container Cloud version.
      	Proxy	Available since 2.5.0 Optional. From the drop-down list, select the proxy server name that you have previously created.
      	SSH Keys	From the drop-down list, select the SSH key name that you have previously added for SSH access to VMs.
      Provider	External Network	Type of the external network in the OpenStack cloud provider.
      	DNS Name Servers	Comma-separated list of the DNS hosts IPs for the OpenStack VMs configuration.
      Kubernetes	Node CIDR	The Kubernetes nodes CIDR block. For example, 10.10.10.0/24.
      	Services CIDR Blocks	The Kubernetes Services CIDR block. For example, 10.233.0.0/18.
      	Pods CIDR Blocks	The Kubernetes Pods CIDR block. For example, 10.233.64.0/18.

   2. Configure StackLight:

      StackLight configuration¶

      Section	Parameter name	Description
      StackLight	Enable Monitoring	Selected by default. Deselect to skip StackLight deployment. Note You can also enable, disable, or configure StackLight parameters after deploying a managed cluster. For details, see Change a cluster configuration or Configure StackLight.
      	Enable Logging	Select to deploy the StackLight logging stack. For details about the logging components, see Reference Architecture: StackLight deployment architecture.
      	HA Mode	Select to enable StackLight monitoring in the HA mode. For the differences between HA and non-HA modes, see Reference Architecture: StackLight deployment architecture.
      Elasticsearch	Retention Time	The Elasticsearch logs retention period in Logstash.
      	Persistent Volume Claim Size	The Elasticsearch persistent volume claim size.
      Prometheus	Retention Time	The Prometheus database retention period.
      	Retention Size	The Prometheus database retention size.
      	Persistent Volume Claim Size	The Prometheus persistent volume claim size.
      	Enable Watchdog Alert	Select to enable the Watchdog alert that fires as long as the entire alerting pipeline is functional.
      	Custom Alerts	Specify alerting rules for new custom alerts or upload a YAML file in the following exemplary format: - alert: HighErrorRate expr: job:request_latency_seconds:mean5m{job="myjob"} > 0.5 for: 10m labels: severity: page annotations: summary: High request latency For details, see Official Prometheus documentation: Alerting rules. For the list of the predefined StackLight alerts, see Operations Guide: Available StackLight alerts.
      StackLight Email Alerts	Enable Email Alerts	Select to enable the StackLight email alerts.
      	Send Resolved	Select to enable notifications about resolved StackLight alerts.
      	Require TLS	Select to enable transmitting emails through TLS.
      	Email alerts configuration for StackLight	Fill out the following email alerts parameters as required: To - the email address to send notifications to. From - the sender address. SmartHost - the SMTP host through which the emails are sent. Authentication username - the SMTP user name. Authentication password - the SMTP password. Authentication identity - the SMTP identity. Authentication secret - the SMTP secret.
      StackLight Slack Alerts	Enable Slack alerts	Select to enable the StackLight Slack alerts.
      	Send Resolved	Select to enable notifications about resolved StackLight alerts.
      	Slack alerts configuration for StackLight	Fill out the following Slack alerts parameters as required: API URL - The Slack webhook URL. Channel - The channel to send notifications to, for example, #channel-for-alerts.

7. Click Create.

   To view the deployment status, verify the cluster status on the Clusters page. Once the orange blinking dot near the cluster name disappears, the deployment is complete.

8. Proceed with Add a machine.

Add a machine¶

After you create a new OpenStack-based Mirantis Container Cloud managed cluster as described in Create a managed cluster, proceed with adding machines to this cluster using the Container Cloud web UI.

You can also use the instruction below to scale up an existing managed cluster.

To add a machine to an OpenStack-based managed cluster:

1. Log in to the Container Cloud web UI with the writer permissions.

2. Switch to the required project using the Switch Project action icon located on top of the main left-side navigation panel.

3. In the Clusters tab, click the required cluster name. The cluster page with Machines list opens.

4. On the cluster page, click Create Machine.

5. Fill out the form with the following parameters as required:

   Container Cloud machine configuration¶

   Parameter	Description
   Count	Specify the number of machines to create. The required minimum number of machines is three for the manager nodes HA and two for the Container Cloud workloads. Select Manager or Worker to create a Kubernetes manager or worker node.
   Flavor	From the drop-down list, select the required hardware configuration for the machine. The list of available flavors corresponds to the one in your OpenStack environment. For the hardware requirements, see: Reference Architecture: Requirements for an OpenStack-based cluster.
   Image	From the drop-down list, select the cloud image with Ubuntu 18.04. If you do not have this image in the list, add it to your OpenStack environment using the Horizon web UI by downloading the image from the Ubuntu official website.
   Availability zone	From the drop-down list, select the availability zone from which the new machine will be launched.
   Node Labels	Select the required node labels for the machine to run certain components on a specific node. For example, for the StackLight nodes that run Elasticsearch and require more resources than a standard node, select the StackLight label. The list of available node labels is obtained from your current Cluster release. Caution If you deploy StackLight in the HA mode (recommended), add the StackLight label to minimum three nodes. Note You can configure node labels after deploying a machine. On the Machines page, click the More action icon in the last column of the required machine field and select Configure machine.

6. Click Create.

7. Repeat the steps above for the remaining machines.

   You can monitor the machine status in the Managers or Workers columns on the Clusters page. Once the status changes to Ready, the deployment of the managed cluster components on this machine is complete.

   The machine creation starts with the Provision status. During provisioning, the machine is not expected to be accessible since its infrastructure (VM, network, and so on) is being created.

   Other machine statuses are the same as the LCMMachine object states described in Reference Architecture: LCM controller.

8. Verify the status of the cluster nodes as described in Connect to a Mirantis Container Cloud cluster.

Delete a managed cluster¶

Deleting a managed cluster does not require a preliminary deletion of VMs that run on this cluster.

To delete an OpenStack-based managed cluster:

1. Log in to the Mirantis Container Cloud web UI with the writer permissions.

2. Switch to the required project using the Switch Project action icon located on top of the main left-side navigation panel.

3. In the Clusters tab, click the More action icon in the last column of the required cluster and select Delete.

4. Verify the list of machines to be removed. Confirm the deletion.

   Deleting a cluster automatically frees up the resources allocated for this cluster, for example, instances, load balancers, networks, floating IPs.

5. If the cluster deletion hangs and the The cluster is being deleted message does not disappear for a while:

   1. Expand the menu of the tab with your username.

   2. Click Download kubeconfig to download kubeconfig of your management cluster.

   3. Log in to any local machine with kubectl installed.

   4. Copy the downloaded kubeconfig to this machine.

   5. Run the following command:

      kubectl --kubeconfig <KUBECONFIG_PATH> edit -n <PROJECT_NAME> cluster <MANAGED_CLUSTER_NAME>
      

   6. Edit the opened kubeconfig by removing the following lines:

      finalizers:
      - cluster.cluster.k8s.io
      

6. If you are going to remove the associated regional cluster or if you do not plan to reuse the credentials of the deleted cluster, delete them:

   1. In the Credentials tab, verify that the required credentials are not in the In Use status.

   2. Click the Delete credential action icon next to the name of the credentials to be deleted.

   3. Confirm the deletion.

   Warning

   You can delete credentials only after deleting the managed cluster they relate to.

Create a managed cluster¶

This section describes how to create an AWS-based managed cluster using the Mirantis Container Cloud web UI of the AWS-based management cluster.

To create an AWS-based managed cluster:

1. Log in to the Container Cloud web UI with the writer permissions.

2. Switch to the required project using the Switch Project action icon located on top of the main left-side navigation panel.

3. In the SSH Keys tab, click Add SSH Key to upload the public SSH key that will be configured on each AWS instance to provide user access.

4. In the Credentials tab:

   1. Click Add Credential and fill in the required fields to add your AWS credentials.

   2. Verify that the new credentials status is Ready. If the status is Error, hover over the status to determine the reason of the issue.

5. In the Clusters tab, click Create Cluster and fill out the form with the following parameters as required:

   1. Configure general settings and the Kubernetes parameters:

      Managed cluster configuration¶

      Section	Parameter	Description
      General settings	Name	Cluster name
      	Provider	Select AWS
      	Provider credential	From the drop-down list, select the previously created AWS credentials name.
      	Release version	The Container Cloud version.
      	SSH keys	From the drop-down list, select the SSH key name that you have previously added for SSH access to VMs.
      Provider	AWS region	From the drop-down list, select the AWS Region for the managed cluster. For example, us-east-2.
      Kubernetes	Services CIDR blocks	The Kubernetes Services CIDR block. For example, 10.233.0.0/18.
      	Pods CIDR blocks	The Kubernetes Pods CIDR block. For example, 10.233.64.0/18.

   2. Configure StackLight:

      StackLight configuration¶

      Section	Parameter name	Description
      StackLight	Enable Monitoring	Selected by default. Deselect to skip StackLight deployment. Note You can also enable, disable, or configure StackLight parameters after deploying a managed cluster. For details, see Change a cluster configuration or Configure StackLight.
      	Enable Logging	Select to deploy the StackLight logging stack. For details about the logging components, see Reference Architecture: StackLight deployment architecture.
      	HA Mode	Select to enable StackLight monitoring in the HA mode. For the differences between HA and non-HA modes, see Reference Architecture: StackLight deployment architecture.
      Elasticsearch	Retention Time	The Elasticsearch logs retention period in Logstash.
      	Persistent Volume Claim Size	The Elasticsearch persistent volume claim size.
      Prometheus	Retention Time	The Prometheus database retention period.
      	Retention Size	The Prometheus database retention size.
      	Persistent Volume Claim Size	The Prometheus persistent volume claim size.
      	Enable Watchdog Alert	Select to enable the Watchdog alert that fires as long as the entire alerting pipeline is functional.
      	Custom Alerts	Specify alerting rules for new custom alerts or upload a YAML file in the following exemplary format: - alert: HighErrorRate expr: job:request_latency_seconds:mean5m{job="myjob"} > 0.5 for: 10m labels: severity: page annotations: summary: High request latency For details, see Official Prometheus documentation: Alerting rules. For the list of the predefined StackLight alerts, see Operations Guide: Available StackLight alerts.
      StackLight Email Alerts	Enable Email Alerts	Select to enable the StackLight email alerts.
      	Send Resolved	Select to enable notifications about resolved StackLight alerts.
      	Require TLS	Select to enable transmitting emails through TLS.
      	Email alerts configuration for StackLight	Fill out the following email alerts parameters as required: To - the email address to send notifications to. From - the sender address. SmartHost - the SMTP host through which the emails are sent. Authentication username - the SMTP user name. Authentication password - the SMTP password. Authentication identity - the SMTP identity. Authentication secret - the SMTP secret.
      StackLight Slack Alerts	Enable Slack alerts	Select to enable the StackLight Slack alerts.
      	Send Resolved	Select to enable notifications about resolved StackLight alerts.
      	Slack alerts configuration for StackLight	Fill out the following Slack alerts parameters as required: API URL - The Slack webhook URL. Channel - The channel to send notifications to, for example, #channel-for-alerts.

6. Click Create.

   To view the deployment status, verify the cluster status on the Clusters page. Once the orange blinking dot near the cluster name disappears, the deployment is complete.

7. Proceed with Add a machine.

Add a machine¶

After you create a new AWS-based managed cluster as described in Create a managed cluster, proceed with adding machines to this cluster using the Mirantis Container Cloud web UI.

You can also use the instruction below to scale up an existing managed cluster.

To add a machine to an AWS-based managed cluster:

1. Log in to the Container Cloud web UI with the writer permissions.

2. Switch to the required project using the Switch Project action icon located on top of the main left-side navigation panel.

3. In the Clusters tab, click the required cluster name. The cluster page with the Machines list opens.

4. Click Create Machine.

5. Fill out the form with the following parameters as required:

   Container Cloud machine configuration¶

   Parameter	Description
   Count	Specify the number of machines to create. The required minimum number of machines is three for the manager nodes HA and two for the Container Cloud workloads. Select Manager or Worker to create a Kubernetes manager or worker node.
   Instance type	From the drop-down list, select the required AWS instance type. For production deployments, Mirantis recommends: c5d.2xlarge for worker nodes c5d.4xlarge for manager nodes r5.4xlarge for nodes where the StackLight server components run For more details about requirements, see Reference architecture: AWS system requirements.
   AMI ID	From the drop-down list, select the required AMI ID of Ubuntu 18.04. For example, ubuntu-bionic-18.04-amd64-server-20200729.
   Root device size	Select the required root device size, 40 by default.
   Node Labels	Select the required node labels for the machine to run certain components on a specific node. For example, for the StackLight nodes that run Elasticsearch and require more resources than a standard node, select the StackLight label. The list of available node labels is obtained from your current Cluster release. Caution If you deploy StackLight in the HA mode (recommended), add the StackLight label to minimum three nodes. Note You can configure node labels after deploying a machine. On the Machines page, click the More action icon in the last column of the required machine field and select Configure machine.

6. Click Create.

7. Repeat the steps above for the remaining machines.

   You can monitor the machine status in the Managers or Workers columns on the Clusters page. Once the status changes to Ready, the deployment of the managed cluster components on this machine is complete.

   The machine creation starts with the Provision status. During provisioning, the machine is not expected to be accessible since its infrastructure (VM, network, and so on) is being created.

   Other machine statuses are the same as the LCMMachine object states described in Reference Architecture: LCM controller.

8. Verify the status of the cluster nodes as described in Connect to a Mirantis Container Cloud cluster.

Delete a managed cluster¶

Deleting a managed cluster does not require a preliminary deletion of VMs that run on this cluster.

To delete an AWS-based managed cluster:

1. Log in to the Container Cloud web UI with the writer permissions.

2. Switch to the required project using the Switch Project action icon located on top of the main left-side navigation panel.

3. In the Clusters tab, click the More action icon in the last column of the required cluster and select Delete.

4. Verify the list of machines to be removed. Confirm the deletion.

   Deleting a cluster automatically removes the Amazon Virtual Private Cloud (VPC) connected with this cluster and frees up the resources allocated for this cluster, for example, instances, load balancers, networks, floating IPs.

5. If you are going to remove the associated regional cluster or if you do not plan to reuse the credentials of the deleted cluster, delete them:

   1. In the Credentials tab, verify that the required credentials are not in the In Use status.

   2. Click the Delete credential action icon next to the name of the credentials to be deleted.

   3. Confirm the deletion.

   Warning

   You can delete credentials only after deleting the managed cluster they relate to.

Create a managed cluster¶

This section describes how to create a VMWare vSphere-based managed cluster using the Mirantis Container Cloud web UI of the vSphere-based management cluster.

To create a vSphere-based managed cluster:

1. Log in to the Container Cloud web UI with the writer permissions.

2. Switch to the required project using the Switch Project action icon located on top of the main left-side navigation panel.

3. In the SSH Keys tab, click Add SSH Key to upload the public SSH key that will be used for the vSphere VMs creation.

4. In the Credentials tab:

   1. Click Add Credential to add your vSphere credentials. You can either upload your vSphere vsphere.yaml configuration file or fill in the fields manually.

   2. Verify that the new credentials status is Ready. If the status is Error, hover over the status to determine the reason of the issue.

5. In the RHEL Licenses tab, click Add RHEL License and fill out the form with the following parameters:

   RHEL license parameters¶

   Parameter	Description
   RHEL License Name	RHEL license name
   Username	User name to access the RHEL license
   Password	Password to access the RHEL license
   Pool IDs	Optional. Specify the pool IDs for RHEL licenses for Virtual Datacenters. Otherwise, Subscription Manager will select a subscription from the list of available and appropriate for the machines.

6. In the Clusters tab, click Create Cluster and fill out the form with the following parameters as required:

   1. Configure general settings and Kubernetes parameters:

      Managed cluster configuration¶

      Section	Parameter	Description
      General Settings	Name	Cluster name
      	Provider	Select vSphere
      	Provider Credential	From the drop-down list, select the vSphere credentials name that you have previously added.
      	Release Version	The Container Cloud version.
      	Proxy	Available since 2.5.0, Technology Preview Optional. Disregard this field since the feature is not fully integrated yet.
      	SSH Keys	From the drop-down list, select the SSH key name that you have previously added for SSH access to VMs.
      Provider	LB Host IP	The IP address of the load balancer endpoint that will be used to access the Kubernetes API of the new cluster.
      	LB Address Range	The range of IP addresses that can be assigned to load balancers for Kubernetes Services.
      Kubernetes	Node CIDR	The Kubernetes nodes CIDR block. For example, 10.10.10.0/24.
      	Services CIDR Blocks	The Kubernetes Services CIDR block. For example, 10.233.0.0/18.
      	Pods CIDR Blocks	The Kubernetes Pods CIDR block. For example, 10.233.64.0/18.

   2. Configure StackLight:

      StackLight configuration¶

      Section	Parameter name	Description
      StackLight	Enable Monitoring	Selected by default. Deselect to skip StackLight deployment. Note You can also enable, disable, or configure StackLight parameters after deploying a managed cluster. For details, see Change a cluster configuration or Configure StackLight.
      	Enable Logging	Select to deploy the StackLight logging stack. For details about the logging components, see Reference Architecture: StackLight deployment architecture.
      	HA Mode	Select to enable StackLight monitoring in the HA mode. For the differences between HA and non-HA modes, see Reference Architecture: StackLight deployment architecture.
      Elasticsearch	Retention Time	The Elasticsearch logs retention period in Logstash.
      	Persistent Volume Claim Size	The Elasticsearch persistent volume claim size.
      Prometheus	Retention Time	The Prometheus database retention period.
      	Retention Size	The Prometheus database retention size.
      	Persistent Volume Claim Size	The Prometheus persistent volume claim size.
      	Enable Watchdog Alert	Select to enable the Watchdog alert that fires as long as the entire alerting pipeline is functional.
      	Custom Alerts	Specify alerting rules for new custom alerts or upload a YAML file in the following exemplary format: - alert: HighErrorRate expr: job:request_latency_seconds:mean5m{job="myjob"} > 0.5 for: 10m labels: severity: page annotations: summary: High request latency For details, see Official Prometheus documentation: Alerting rules. For the list of the predefined StackLight alerts, see Operations Guide: Available StackLight alerts.
      StackLight Email Alerts	Enable Email Alerts	Select to enable the StackLight email alerts.
      	Send Resolved	Select to enable notifications about resolved StackLight alerts.
      	Require TLS	Select to enable transmitting emails through TLS.
      	Email alerts configuration for StackLight	Fill out the following email alerts parameters as required: To - the email address to send notifications to. From - the sender address. SmartHost - the SMTP host through which the emails are sent. Authentication username - the SMTP user name. Authentication password - the SMTP password. Authentication identity - the SMTP identity. Authentication secret - the SMTP secret.
      StackLight Slack Alerts	Enable Slack alerts	Select to enable the StackLight Slack alerts.
      	Send Resolved	Select to enable notifications about resolved StackLight alerts.
      	Slack alerts configuration for StackLight	Fill out the following Slack alerts parameters as required: API URL - The Slack webhook URL. Channel - The channel to send notifications to, for example, #channel-for-alerts.

7. Click Create.

   To view the deployment status, verify the cluster status on the Clusters page. Once the orange blinking dot near the cluster name disappears, the deployment is complete.

8. Proceed with Add a machine.

Add a machine¶

After you create a new VMWare vSphere-based Mirantis Container Cloud managed cluster as described in Create a managed cluster, proceed with adding machines to this cluster using the Container Cloud web UI.

You can also use the instruction below to scale up an existing managed cluster.

To add a machine to a vSphere-based managed cluster:

1. Log in to the Container Cloud web UI with the writer permissions.

2. Switch to the required project using the Switch Project action icon located on top of the main left-side navigation panel.

3. In the Clusters tab, click the required cluster name. The cluster page with Machines list opens.

4. On the cluster page, click Create Machine.

5. Fill out the form with the following parameters as required:

   Container Cloud machine configuration¶

   Parameter	Description
   Count	Number of machines to create. The required minimum number of machines is three for the manager nodes HA and two for the Container Cloud workloads. Select Manager or Worker to create a Kubernetes manager or worker node.
   Template Path	Path to the prepared OVF template.
   SSH Username	SSH user name to access the node. Defaults to cloud-user.
   RHEL License	From the drop-down list, select the RHEL license that you previously added for the cluster being deployed.
   Node Labels	Select the required node labels for the machine to run certain components on a specific node. For example, for the StackLight nodes that run Elasticsearch and require more resources than a standard node, select the StackLight label. The list of available node labels is obtained from your current Cluster release. Caution If you deploy StackLight in the HA mode (recommended), add the StackLight label to minimum three nodes. Note You can configure node labels after deploying a machine. On the Machines page, click the More action icon in the last column of the required machine field and select Configure machine.

6. Click Create.

7. Repeat the steps above for the remaining machines.

   You can monitor the machine status in the Managers or Workers columns on the Clusters page. Once the status changes to Ready, the deployment of the managed cluster components on this machine is complete.

   The machine creation starts with the Provision status. During provisioning, the machine is not expected to be accessible since its infrastructure (VM, network, and so on) is being created.

   Other machine statuses are the same as the LCMMachine object states described in Reference Architecture: LCM controller.

8. Verify the status of the cluster nodes as described in Connect to a Mirantis Container Cloud cluster.

Delete a managed cluster¶

Deleting a managed cluster does not require a preliminary deletion of VMs that run on this cluster.

To delete a VMWare vSphere-based managed cluster:

1. Log in to the Mirantis Container Cloud web UI with the writer permissions.

2. Switch to the required project using the Switch Project action icon located on top of the main left-side navigation panel.

3. In the Clusters tab, click the More action icon in the last column of the required cluster and select Delete.

4. Verify the list of machines to be removed. Confirm the deletion.

5. Deleting a cluster automatically turns the machines off. Therefore, clean up the hosts manually in the vSphere web UI. The machines will be automatically released from the RHEL subscription.

6. If you are going to remove the associated regional cluster or if you do not plan to reuse the credentials of the deleted cluster, delete them:

   1. In the Credentials tab, verify that the required credentials are not in the In Use status.

   2. Click the Delete credential action icon next to the name of the credentials to be deleted.

   3. Confirm the deletion.

   Warning

   You can delete credentials only after deleting the managed cluster they relate to.

Configure IAM CLI¶

The iamctl command-line interface uses the iamctl.yaml configuration file to interact with IAM.

To create the IAM CLI configuration file:

1. Log in to the management cluster.

2. Change the directory to one of the following:

   • $HOME/.iamctl

   • $HOME

   • $HOME/etc

   • /etc/iamctl

3. Create iamctl.yaml with the following exemplary parameters and values that correspond to your deployment:

   server: <IAM_API_ADDRESS>
   timeout: 60
   verbose: 99 # Verbosity level, from 0 to 99
   
   tls:
       enabled: true
       ca: <PATH_TO_CA_BUNDLE>
   
   auth:
       issuer: <IAM_REALM_IN_KEYCLOAK>
       ca: <PATH_TO_CA_BUNDLE>
       client_id: iam
       client_secret:
   

   The <IAM_REALM_IN_KEYCLOAK> value has the <keycloak-url>/auth/realms/<realm-name> format, where <realm-name> defaults to iam.

Available IAM CLI commands¶

Using iamctl, you can perform different role-based access control operations in your managed cluster. For example:

• Grant or revoke access to a managed cluster and a specific user for troubleshooting

• Grant or revoke access to a Mirantis Container Cloud project that contains several managed clusters

• Create or delete tokens for the Container Cloud services with a specific set of grants as well as identify when a service token was used the last time

The iamctl command-line interface contains the following set of commands:

• General commands

• Account information commands

• Scope commands

• Role commands

• Grant commands

• Service token commands

• User commands

The following tables describe the iamctl commands with their descriptions.

+---------------+--------------------------+
|     NAME      |   DESCRIPTION            |
+---------------+--------------------------+
| m:iam         | IAM scope                |
| m:kaas        | Container Cloud scope    |
| m:k8s:managed |                          |
| m:k8s         | Kubernetes scope         |
| m:cloud       | Cloud scope              |
+---------------+--------------------------+

+--------+--------+---------------+
| SCOPE  |  ROLE  |   GRANT FQN   |
+--------+--------+---------------+
| m:iam  | admin  | m:iam@admin   |
| m:sl   | viewer | m:sl@viewer   |
| m:kaas | writer | m:kaas@writer |
+--------+--------+---------------+

Alert dependencies¶

Using alert inhibition rules, Alertmanager decreases alert noise by suppressing dependent alerts notifications to provide a clearer view on the cloud status and simplify troubleshooting. Alert inhibition rules are enabled by default.

The following table describes the dependency between alerts. Once an alert from the Alert column raises, the alert from the Silences column will be suppressed with the Inhibited status in the Alertmanager web UI.

Alertmanager¶

This section describes the alerts for the Alertmanager service.

• AlertmanagerFailedReload

• AlertmanagerMembersInconsistent

• AlertmanagerNotificationFailureWarning

• AlertmanagerAlertsInvalidWarning

Calico¶

This section describes the alerts for Calico.

• CalicoDataplaneFailuresHigh

• CalicoDataplaneAddressMsgBatchSizeHigh

• CalicoDatapaneIfaceMsgBatchSizeHigh

• CalicoIPsetErrorsHigh

• CalicoIptablesSaveErrorsHigh

• CalicoIptablesRestoreErrorsHigh

Ceph¶

This section describes the alerts for the Ceph cluster.

• CephClusterHealthMinor

• CephClusterHealthCritical

• CephMonQuorumAtRisk

• CephOSDDownMinor

• CephOSDDiskNotResponding

• CephOSDDiskUnavailable

• CephClusterFullWarning

• CephClusterFullCritical

• CephOSDPgNumTooHighWarning

• CephOSDPgNumTooHighCritical

• CephMonHighNumberOfLeaderChanges

• CephNodeDown

• CephOSDVersionMismatch

• CephMonVersionMismatch

• CephPGInconsistent

• CephPGUndersized