Install MKE¶

Hardware and software requirements¶

You can install MKE on-premises or on a cloud provider. Common requirements:

• Mirantis Container Runtime
• Linux kernel version 3.10 or higher. For debugging purposes, it is suggested to match the host OS kernel versions as close as possible.
• A static IP address for each node in the cluster
• User namespaces should not be configured on any node. This function is not currently supported by MKE.

Ports used¶

When installing MKE on a host, a series of ports need to be opened to incoming traffic. Each of these ports will expect incoming traffic from a set of hosts, indicated as the “Scope” of that port. The three scopes are: - External: Traffic arrives from outside the cluster through end-user interaction. - Internal: Traffic arrives from other hosts in the same cluster. - Self: Traffic arrives to that port only from processes on the same host.

Make sure the following ports are open for incoming traffic on the respective host types:

Disable CLOUD_NETCONFIG_MANAGE for SLES 15¶

For SUSE Linux Enterprise Server 15 (SLES 15) installations, you must disable CLOUD_NETCONFIG_MANAGE prior to installing MKE.

1. In the network interface configuration file, `/etc/sysconfig/network/ifcfg-eth0`, set
```
CLOUD_NETCONFIG_MANAGE="no"
```
2. Run `service network restart`.

Enable ESP traffic¶

For overlay networks with encryption to work, you need to ensure that IP protocol 50 (Encapsulating Security Payload) traffic is allowed.

Enable IP-in-IP traffic¶

The default networking plugin for MKE is Calico, which uses IP Protocol Number 4 for IP-in-IP encapsulation.

If you’re deploying to AWS or another cloud provider, enable IP-in-IP traffic for your cloud provider’s security group.

Enable connection tracking on the loopback interface for SLES¶

Calico’s Kubernetes controllers can’t reach the Kubernetes API server unless connection tracking is enabled on the loopback interface. SLES disables connection tracking by default.

On each node in the cluster:

sudo mkdir -p /etc/sysconfig/SuSEfirewall2.d/defaults
echo FW_LO_NOTRACK=no | sudo tee /etc/sysconfig/SuSEfirewall2.d/defaults/99-docker.cfg
sudo SuSEfirewall2 start

Timeout settings¶

Make sure the networks you’re using allow the MKE components enough time to communicate before they time out.

Time Synchronization¶

In distributed systems like MKE, time synchronization is critical to ensure proper operation. As a best practice to ensure consistency between the engines in a MKE cluster, all engines should regularly synchronize time with a Network Time Protocol (NTP) server. If a server’s clock is skewed, unexpected behavior may cause poor performance or even failures.

Compatibility and maintenance lifecycle¶

Docker Enterprise is a software subscription that includes three products:

• Mirantis Container Runtime with enterprise-grade support
• Mirantis Secure Registry
• Mirantis Kubernetes Engine

System requirements¶

Before installing MKE, make sure that all nodes (physical or virtual machines) that you’ll manage with MKE:

• Comply with the system requirements, and
• Are running the same version of Mirantis Container Runtime.

Hostname strategy¶

MKE requires Docker Enterprise. Before installing Docker Enterprise on your cluster nodes, you should plan for a common hostname strategy.

Decide if you want to use short hostnames, like engine01, or Fully Qualified Domain Names (FQDN), like node01.company.example.com. Whichever you choose, confirm your naming strategy is consistent across the cluster, because MKE and Mirantis Container Runtime use hostnames.

For example, if your cluster has three hosts, you can name them:

node1.company.example.com
node2.company.example.com
node3.company.example.com

Static IP addresses¶

MKE requires each node on the cluster to have a static IPv4 address. Before installing MKE, ensure your network and nodes are configured to support this.

Avoid IP range conflicts¶

The following table lists recommendations to avoid IP range conflicts.

{
   "default-address-pools": [
         {"base":"172.17.0.0/16","size":16}, <-- docker0
         {"base":"172.18.0.0/16","size":16}, <-- docker_gwbridge
         {"base":"172.19.0.0/16","size":16},
         {"base":"172.20.0.0/16","size":16},
         {"base":"172.21.0.0/16","size":16},
         {"base":"172.22.0.0/16","size":16},
         {"base":"172.23.0.0/16","size":16},
         {"base":"172.24.0.0/16","size":16},
         {"base":"172.25.0.0/16","size":16},
         {"base":"172.26.0.0/16","size":16},
         {"base":"172.27.0.0/16","size":16},
         {"base":"172.28.0.0/16","size":16},
         {"base":"172.29.0.0/16","size":16},
         {"base":"172.30.0.0/16","size":16},
         {"base":"192.168.0.0/16","size":20}
   ]
}

{
   "default-address-pools": [
         {"base":"172.17.0.0/16","size":16}, <-- Modify this value
         {"base":"172.18.0.0/16","size":16},
         {"base":"172.19.0.0/16","size":16},
         {"base":"172.20.0.0/16","size":16},
         {"base":"172.21.0.0/16","size":16},
         {"base":"172.22.0.0/16","size":16},
         {"base":"172.23.0.0/16","size":16},
         {"base":"172.24.0.0/16","size":16},
         {"base":"172.25.0.0/16","size":16},
         {"base":"172.26.0.0/16","size":16},
         {"base":"172.27.0.0/16","size":16},
         {"base":"172.28.0.0/16","size":16},
         {"base":"172.29.0.0/16","size":16},
         {"base":"172.30.0.0/16","size":16},
         {"base":"192.168.0.0/16","size":20}
   ]
}

{
  "fixed-cidr": "172.17.0.0/16",
}

{
  "bip": "172.17.0.0/16",
}

{
    "default-address-pools": [
       {"base":"172.17.0.0/16","size":16},
       {"base":"172.18.0.0/16","size":16}, <-- Modify this value
       {"base":"172.19.0.0/16","size":16},
       {"base":"172.20.0.0/16","size":16},
       {"base":"172.21.0.0/16","size":16},
       {"base":"172.22.0.0/16","size":16},
       {"base":"172.23.0.0/16","size":16},
       {"base":"172.24.0.0/16","size":16},
       {"base":"172.25.0.0/16","size":16},
       {"base":"172.26.0.0/16","size":16},
       {"base":"172.27.0.0/16","size":16},
       {"base":"172.28.0.0/16","size":16},
       {"base":"172.29.0.0/16","size":16},
       {"base":"172.30.0.0/16","size":16},
       {"base":"192.168.0.0/16","size":20}
   ]
}

Avoid firewall conflicts¶

For SUSE Linux Enterprise Server 12 SP2 (SLES12), the FW_LO_NOTRACK flag is turned on by default in the openSUSE firewall. This speeds up packet processing on the loopback interface, and breaks certain firewall setups that need to redirect outgoing packets via custom rules on the local machine.

To turn off the FW_LO_NOTRACK option, edit the /etc/sysconfig/SuSEfirewall2 file and set FW_LO_NOTRACK="no". Save the file and restart the firewall or reboot.

For SUSE Linux Enterprise Server 12 SP3, the default value for FW_LO_NOTRACK was changed to no.

For Red Hat Enterprise Linux (RHEL) 8, if firewalld is running and FirewallBackend=nftables is set in /etc/firewalld/firewalld.conf, change this to FirewallBackend=iptables, or you can explicitly run the following commands to allow traffic to enter the default bridge (docker0) network:

firewall-cmd --permanent --zone=trusted --add-interface=docker0
firewall-cmd --reload

Time synchronization¶

In distributed systems like MKE, time synchronization is critical to ensure proper operation. As a best practice to ensure consistency between the engines in a MKE cluster, all engines should regularly synchronize time with a Network Time Protocol (NTP) server. If a host node’s clock is skewed, unexpected behavior may cause poor performance or even failures.

Load balancing strategy¶

MKE doesn’t include a load balancer. You can configure your own load balancer to balance user requests across all manager nodes.

If you plan to use a load balancer, you need to decide whether you’ll add the nodes to the load balancer using their IP addresses or their FQDNs. Whichever you choose, be consistent across nodes. When this is decided, take note of all IPs or FQDNs before starting the installation.

Load balancing MKE and MSR¶

By default, MKE and MSR both use port 443. If you plan on deploying MKE and MSR, your load balancer needs to distinguish traffic between the two by IP address or port number.

• If you want to configure your load balancer to listen on port 443:
  • Use one load balancer for MKE and another for MSR.
  • Use the same load balancer with multiple virtual IPs.
• Configure your load balancer to expose MKE or MSR on a port other than 443.

If you want to install MKE in a high-availability configuration that uses a load balancer in front of your MKE controllers, include the appropriate IP address and FQDN of the load balancer’s VIP by using one or more --san flags in the MKE install command or when you’re asked for additional SANs in interactive mode.

Use an external Certificate Authority¶

You can customize MKE to use certificates signed by an external Certificate Authority. When using your own certificates, you need to have a certificate bundle that has:

• A ca.pem file with the root CA public certificate,
• A cert.pem file with the server certificate and any intermediate CA public certificates. This certificate should also have SANs for all addresses used to reach the MKE manager,
• A key.pem file with server private key.

You can have a certificate for each manager, with a common SAN. For example, on a three-node cluster, you can have:

• node1.company.example.org with SAN ucp.company.org
• node2.company.example.org with SAN ucp.company.org
• node3.company.example.org with SAN ucp.company.org

You can also install MKE with a single externally-signed certificate for all managers, rather than one for each manager node. In this case, the certificate files are copied automatically to any new manager nodes joining the cluster or being promoted to a manager role.

Step 1: Validate the system requirements¶

The first step to installing MKE is ensuring that your infrastructure has all of the requirements MKE needs to run. Also, you need to ensure that all nodes, physical and virtual, are running the same version of Docker Enterprise.

Step 2: Install Docker Enterprise on all nodes¶

MKE is a containerized application that requires the commercially supported Mirantis Container Runtime to run.

Install Docker Enterprise on each host that you plan to manage with MKE. View the supported platforms and click on your platform to get platform-specific instructions for installing Docker Enterprise.

Make sure you install the same Docker Enterprise version on all the nodes. Also, if you’re creating virtual machine templates with Docker Enterprise already installed, make sure the /etc/docker/key.json file is not included in the virtual machine image. When provisioning the virtual machine, restart the Docker daemon to generate a new /etc/docker/key.json file.

Step 3: Customize named volumes¶

Skip this step if you want to use the defaults provided by MKE.

MKE uses named volumes to persist data. If you want to customize the drivers used to manage these volumes, you can create the volumes before installing MKE. When you install MKE, the installer will notice that the volumes already exist, and it will start using them.

If these volumes don’t exist, they’ll be automatically created when installing MKE.

Step 4: Install MKE¶

To install MKE, you use the docker/ucp image, which has commands to install and manage MKE.

Make sure you follow the MKE System requirements for opening networking ports. Ensure that your hardware or software firewalls are open appropriately or disabled.

1. Use ssh to log in to the host where you want to install MKE.

2. Run the following command:

   # Pull the latest version of UCP
   docker image pull docker/ucp:3.2.5
   
   # Install UCP
   docker container run --rm -it --name ucp \
     -v /var/run/docker.sock:/var/run/docker.sock \
     docker/ucp:3.2.5 install \
     --host-address <node-ip-address> \
     --interactive
   

   This runs the install command in interactive mode, so that you’re prompted for any necessary configuration values. To find what other options are available in the install command, including how to install MKE on a system with SELinux enabled, check the reference documentation.

Step 5: License your installation¶

Now that MKE is installed, you need to license it. To use MKE, you are required to have a Docker Enterprise subscription, or you can test the platform with a free trial license.

1. Go to Docker Hub to get a free trial license.
2. In your browser, navigate to the MKE web UI, log in with your administrator credentials and upload your license. Navigate to the Admin Settings page and in the left pane, click License.
3. Click Upload License and navigate to your license (.lic) file. When you’re finished selecting the license, MKE updates with the new settings.

Step 6: Join manager nodes¶

To make your Docker swarm and MKE fault-tolerant and highly available, you can join more manager nodes to it. Manager nodes are the nodes in the swarm that perform the orchestration and swarm management tasks, and dispatch tasks for worker nodes to execute.

To join manager nodes to the swarm,

1. In the MKE web UI, navigate to the Nodes page, and click the Add Node button to add a new node.
2. In the Add Node page, check Add node as a manager to turn this node into a manager and replicate MKE for high-availability.
3. If you want to customize the network and port where the new node listens for swarm management traffic, click Use a custom listen address. Enter the IP address and port for the node to listen for inbound cluster management traffic. The format is interface:port or ip:port. The default is 0.0.0.0:2377.
4. If you want to customize the network and port that the new node advertises to other swarm members for API access, click Use a custom advertise address and enter the IP address and port. By default, this is also the outbound address used by the new node to contact MKE. The joining node should be able to contact itself at this address. The format is interface:port or ip:port.
5. Click the copy icon to copy the docker swarm join command that nodes use to join the swarm.
6. For each manager node that you want to join to the swarm, log in using ssh and run the join command that you copied. After the join command completes, the node appears on the Nodes page in the MKE web UI.

Step 7: Join worker nodes¶

To add more computational resources to your swarm, you can join worker nodes. These nodes execute tasks assigned to them by the manager nodes. Follow the same steps as before, but don’t check the Add node as a manager option.

Versions available¶

Use a computer with internet access to download the MKE package from the following links.

Download the offline package¶

You can also use these links to get the MKE package from the command line:

$ wget <mke-package-url> -O ucp.tar.gz

Now that you have the package in your local machine, you can transfer it to the machines where you want to install MKE.

For each machine that you want to manage with MKE:

1. Copy the MKE package to the machine.

   $ scp ucp.tar.gz <user>@<host>
   

2. Use ssh to log in to the hosts where you transferred the package.

3. Load the MKE images.

   Once the package is transferred to the hosts, you can use the docker load command, to load the Docker images from the tar archive:

   $ docker load -i ucp.tar.gz
   

Follow the same steps for the MSR binaries.

Install MKE¶

Now that the offline hosts have all the images needed to install MKE, you can install MKE on one of the manager nodes.

Install MKE on AWS¶

Mirantis Kubernetes Engine (MKE) can be installed on top of AWS without any customisation following the MKE install documentation. Therefore this document is optional, however if you are deploying Kubernetes workloads with MKE and want to leverage the AWS kubernetes cloud provider, which provides dynamic volume and loadbalancer provisioning then you should follow this guide. This guide is not required if you are only deploying swarm workloads.

The requirements for installing MKE on AWS are included in the following sections:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [ "ec2:*" ],
      "Resource": [ "*" ]
    },
    {
      "Effect": "Allow",
      "Action": [ "elasticloadbalancing:*" ],
      "Resource": [ "*" ]
    },
    {
      "Effect": "Allow",
      "Action": [ "route53:*" ],
      "Resource": [ "*" ]
    },
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [ "arn:aws:s3:::kubernetes-*" ]
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [ "arn:aws:s3:::kubernetes-*" ]
    },
    {
      "Effect": "Allow",
      "Action": "ec2:Describe*",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "ec2:AttachVolume",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "ec2:DetachVolume",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [ "route53:*" ],
      "Resource": [ "*" ]
    }
}

$ docker container run --rm -it \
  --name ucp \
  --volume /var/run/docker.sock:/var/run/docker.sock \
  docker/ucp:3.2.5 install \
  --host-address <mke-ip> \
  --cloud-provider aws \
  --interactive

Install MKE on Azure¶

Mirantis Kubernetes Engine (MKE) closely integrates with Microsoft Azure for its Kubernetes Networking and Persistent Storage feature set. MKE deploys the Calico CNI provider. In Azure, the Calico CNI leverages the Azure networking infrastructure for data path networking and the Azure IPAM for IP address management. There are infrastructure prerequisites required prior to MKE installation for the Calico / Azure integration.

{
    "cloud":"AzurePublicCloud",
    "tenantId": "***",
    "subscriptionId": "***",
    "aadClientId": "***",
    "aadClientSecret": "***",
    "resourceGroup": "***",
    "location": "***",
    "subnetName": "***",
    "securityGroupName": "***",
    "vnetName": "***",
    "useInstanceMetadata": true
}

"networkProfile": {
  "networkInterfaceConfigurations": [
    {
      "name": "[variables('nicName')]",
      "properties": {
        "ipConfigurations": [
          {
            "name": "[variables('ipConfigName1')]",
            "properties": {
              "primary": "true",
              "subnet": {
                "id": "[concat('/subscriptions/', subscription().subscriptionId,'/resourceGroups/', resourceGroup().name, '/providers/Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'), '/subnets/', variables('subnetName'))]"
              },
              "loadBalancerBackendAddressPools": [
                {
                  "id": "[concat('/subscriptions/', subscription().subscriptionId,'/resourceGroups/', resourceGroup().name, '/providers/Microsoft.Network/loadBalancers/', variables('loadBalancerName'), '/backendAddressPools/', variables('bePoolName'))]"
                }
              ],
              "loadBalancerInboundNatPools": [
                {
                  "id": "[concat('/subscriptions/', subscription().subscriptionId,'/resourceGroups/', resourceGroup().name, '/providers/Microsoft.Network/loadBalancers/', variables('loadBalancerName'), '/inboundNatPools/', variables('natPoolName'))]"
                }
              ]
            }
          },
          {
            "name": "[variables('ipConfigName2')]",
            "properties": {
              "subnet": {
                "id": "[concat('/subscriptions/', subscription().subscriptionId,'/resourceGroups/', resourceGroup().name, '/providers/Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'), '/subnets/', variables('subnetName'))]"
              }
            }
          }
          .
          .
          .
          {
            "name": "[variables('ipConfigName32')]",
            "properties": {
              "subnet": {
                "id": "[concat('/subscriptions/', subscription().subscriptionId,'/resourceGroups/', resourceGroup().name, '/providers/Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'), '/subnets/', variables('subnetName'))]"
              }
            }
          }
        ],
        "primary": "true"
      }
    }
  ]
}

docker container run --rm -it \
  --name ucp \
  --volume /var/run/docker.sock:/var/run/docker.sock \
  docker/ucp:3.2.5 install \
  --host-address <mke-ip> \
  --pod-cidr <ip-address-range> \
  --cloud-provider Azure \
  --interactive

Plan the upgrade¶

Before upgrading to a new version of MKE, check the Docker Enterprise Release Notes. There you’ll find information about new features, breaking changes, and other relevant information for upgrading to a particular version.

As part of the upgrade process, you’ll upgrade the Mirantis Container Runtime installed on each node of the cluster to version 19.03 or higher. You should plan for the upgrade to take place outside of business hours, to ensure there’s minimal impact to your users.

Also, don’t make changes to MKE configurations while you’re upgrading it. This can lead to misconfigurations that are difficult to troubleshoot.

Back up your cluster¶

Before starting an upgrade, make sure that your cluster is healthy. If a problem occurs, this makes it easier to find and troubleshoot it.

Create a backup of your cluster. This allows you to recover if something goes wrong during the upgrade process.

Upgrade Mirantis Container Runtime¶

For each node that is part of your cluster, upgrade the Mirantis Container Runtime installed on that node to Mirantis Container Runtime version 19.03 or higher. Be sure to install the Docker Enterprise Edition.

Starting with the manager nodes, and then worker nodes:

1. Log into the node using ssh. #. Upgrade the Mirantis Container Runtime to version 18.09.0 or higher. See Upgrade Docker EE.
2. Make sure the node is healthy.

Upgrade MKE¶

When upgrading MKE to version 3.2.5, you can choose from different upgrade workflows:

• Automated, in-place cluster upgrade: Performed on any manager node. Automatically upgrades the entire cluster.
• Manual cluster upgrade: Performed using the CLI. Automatically upgrades manager nodes and allows you to control the upgrade order of worker nodes. This type of upgrade is more advanced than the automated, in-place cluster upgrade.
  • Upgrade existing nodes in place: Performed using the CLI. Automatically upgrades manager nodes and allows you to control the order of worker node upgrades.
  • Replace all worker nodes using blue-green deployment: Performed using the CLI. This type of upgrade allows you to stand up a new cluster in parallel to the current code and cut over when complete. This type of upgrade allows you to join new worker nodes, schedule workloads to run on new nodes, pause, drain, and remove old worker nodes in batches of multiple nodes rather than one at a time, and shut down servers to remove worker nodes. This type of upgrade is the most advanced.

$ docker container run --rm -it \
  --name ucp \
  --volume /var/run/docker.sock:/var/run/docker.sock \
  docker/ucp:3.2.5 \
  upgrade \
  --interactive

$ docker container run --rm -it \
  --name ucp \
  --volume /var/run/docker.sock:/var/run/docker.sock \
  docker/ucp:3.2.5 \
  upgrade \
  --manual-worker-upgrade \
  --interactive

$ docker node update --label-rm com.docker.ucp.upgrade-hold <node name or id>

$ docker swarm join --token SWMTKN-<YOUR TOKEN> <manager ip>:2377

Versions available¶

Use a computer with internet access to download the MKE package from the following links.

Download the offline package¶

You can also use these links to get the MKE package from the command line:

$ wget <mke-package-url> -O ucp.tar.gz

Now that you have the package in your local machine, you can transfer it to the machines where you want to upgrade MKE.

For each machine that you want to manage with MKE:

1. Copy the offline package to the machine.

   $ scp ucp.tar.gz <user>@<host>
   

2. Use ssh to log in to the hosts where you transferred the package.

3. Load the MKE images.

   Once the package is transferred to the hosts, you can use the docker load command, to load the Docker images from the tar archive:

   $ docker load -i ucp.tar.gz
   

Upgrade MKE¶

Now that the offline hosts have all the images needed to upgrade MKE, you can upgrade MKE.

Swarm mode CA¶

After uninstalling MKE, the nodes in your cluster will still be in swarm mode, but you can’t join new nodes until you reinstall MKE, because swarm mode relies on MKE to provide the CA certificates that allow nodes in the cluster to identify one another. Also, since swarm mode is no longer controlling its own certificates, if the certificates expire after you uninstall MKE, the nodes in the swarm won’t be able to communicate at all. To fix this, either reinstall MKE before the certificates expire or disable swarm mode by running docker swarm leave --force on every node.

Restore IP tables¶

When you install MKE, the Calico network plugin changes the host’s IP tables. When you uninstall MKE, the IP tables aren’t reverted to their previous state. After you uninstall MKE, restart the node to restore its IP tables.