Install UCP¶

Hardware and software requirements¶

You can install UCP on-premises or on a cloud provider. Common requirements:

• Docker Engine - Enterprise
• Linux kernel version 3.10 or higher. For debugging purposes, it is suggested to match the host OS kernel versions as close as possible.
• A static IP address for each node in the cluster
• User namespaces should not be configured on any node. This function is not currently supported by UCP.

Ports used¶

When installing UCP on a host, a series of ports need to be opened to incoming traffic. Each of these ports will expect incoming traffic from a set of hosts, indicated as the “Scope” of that port. The three scopes are: - External: Traffic arrives from outside the cluster through end-user interaction. - Internal: Traffic arrives from other hosts in the same cluster. - Self: Traffic arrives to that port only from processes on the same host.

Make sure the following ports are open for incoming traffic on the respective host types:

Disable CLOUD_NETCONFIG_MANAGE for SLES 15¶

For SUSE Linux Enterprise Server 15 (SLES 15) installations, you must disable CLOUD_NETCONFIG_MANAGE prior to installing UCP.

1. In the network interface configuration file, `/etc/sysconfig/network/ifcfg-eth0`, set
```
CLOUD_NETCONFIG_MANAGE="no"
```
2. Run `service network restart`.

Enable ESP traffic¶

For overlay networks with encryption to work, you need to ensure that IP protocol 50 (Encapsulating Security Payload) traffic is allowed.

Enable IP-in-IP traffic¶

The default networking plugin for UCP is Calico, which uses IP Protocol Number 4 for IP-in-IP encapsulation.

If you’re deploying to AWS or another cloud provider, enable IP-in-IP traffic for your cloud provider’s security group.

Enable connection tracking on the loopback interface for SLES¶

Calico’s Kubernetes controllers can’t reach the Kubernetes API server unless connection tracking is enabled on the loopback interface. SLES disables connection tracking by default.

On each node in the cluster:

sudo mkdir -p /etc/sysconfig/SuSEfirewall2.d/defaults
echo FW_LO_NOTRACK=no | sudo tee /etc/sysconfig/SuSEfirewall2.d/defaults/99-docker.cfg
sudo SuSEfirewall2 start

Timeout settings¶

Make sure the networks you’re using allow the UCP components enough time to communicate before they time out.

Time Synchronization¶

In distributed systems like UCP, time synchronization is critical to ensure proper operation. As a best practice to ensure consistency between the engines in a UCP cluster, all engines should regularly synchronize time with a Network Time Protocol (NTP) server. If a server’s clock is skewed, unexpected behavior may cause poor performance or even failures.

Compatibility and maintenance lifecycle¶

Docker Enterprise is a software subscription that includes three products:

• Docker Engine - Enterprise with enterprise-grade support
• Docker Trusted Registry
• Universal Control Plane

Version compatibility¶

UCP 3.1.8 requires minimum versions of the following Docker components:

• Docker Enterprise Engine 18.09.0-ee-1 or higher
• DTR 2.6 or higher

System requirements¶

Before installing UCP, make sure that all nodes (physical or virtual machines) that you’ll manage with UCP:

• Comply with the system requirements, and
• Are running the same version of Docker Engine.

Hostname strategy¶

UCP requires Docker Enterprise. Before installing Docker Enterprise on your cluster nodes, you should plan for a common hostname strategy.

Decide if you want to use short hostnames, like engine01, or Fully Qualified Domain Names (FQDN), like node01.company.example.com. Whichever you choose, confirm your naming strategy is consistent across the cluster, because Docker Engine and UCP use hostnames.

For example, if your cluster has three hosts, you can name them:

node1.company.example.com
node2.company.example.com
node3.company.example.com

Static IP addresses¶

UCP requires each node on the cluster to have a static IPv4 address. Before installing UCP, ensure your network and nodes are configured to support this.

Avoid IP range conflicts¶

The following table lists recommendations to avoid IP range conflicts.

{
   "default-address-pools": [
         {"base":"172.17.0.0/16","size":16}, <-- docker0
         {"base":"172.18.0.0/16","size":16}, <-- docker_gwbridge
         {"base":"172.19.0.0/16","size":16},
         {"base":"172.20.0.0/16","size":16},
         {"base":"172.21.0.0/16","size":16},
         {"base":"172.22.0.0/16","size":16},
         {"base":"172.23.0.0/16","size":16},
         {"base":"172.24.0.0/16","size":16},
         {"base":"172.25.0.0/16","size":16},
         {"base":"172.26.0.0/16","size":16},
         {"base":"172.27.0.0/16","size":16},
         {"base":"172.28.0.0/16","size":16},
         {"base":"172.29.0.0/16","size":16},
         {"base":"172.30.0.0/16","size":16},
         {"base":"192.168.0.0/16","size":20}
   ]
}

{
   "default-address-pools": [
         {"base":"172.17.0.0/16","size":16}, <-- Modify this value
         {"base":"172.18.0.0/16","size":16},
         {"base":"172.19.0.0/16","size":16},
         {"base":"172.20.0.0/16","size":16},
         {"base":"172.21.0.0/16","size":16},
         {"base":"172.22.0.0/16","size":16},
         {"base":"172.23.0.0/16","size":16},
         {"base":"172.24.0.0/16","size":16},
         {"base":"172.25.0.0/16","size":16},
         {"base":"172.26.0.0/16","size":16},
         {"base":"172.27.0.0/16","size":16},
         {"base":"172.28.0.0/16","size":16},
         {"base":"172.29.0.0/16","size":16},
         {"base":"172.30.0.0/16","size":16},
         {"base":"192.168.0.0/16","size":20}
   ]
}

{
  "fixed-cidr": "172.17.0.0/16",
}

{
  "bip": "172.17.0.0/16",
}

{
    "default-address-pools": [
       {"base":"172.17.0.0/16","size":16},
       {"base":"172.18.0.0/16","size":16}, <-- Modify this value
       {"base":"172.19.0.0/16","size":16},
       {"base":"172.20.0.0/16","size":16},
       {"base":"172.21.0.0/16","size":16},
       {"base":"172.22.0.0/16","size":16},
       {"base":"172.23.0.0/16","size":16},
       {"base":"172.24.0.0/16","size":16},
       {"base":"172.25.0.0/16","size":16},
       {"base":"172.26.0.0/16","size":16},
       {"base":"172.27.0.0/16","size":16},
       {"base":"172.28.0.0/16","size":16},
       {"base":"172.29.0.0/16","size":16},
       {"base":"172.30.0.0/16","size":16},
       {"base":"192.168.0.0/16","size":20}
   ]
}

Avoid firewall conflicts¶

For SUSE Linux Enterprise Server 12 SP2 (SLES12), the FW_LO_NOTRACK flag is turned on by default in the openSUSE firewall. This speeds up packet processing on the loopback interface, and breaks certain firewall setups that need to redirect outgoing packets via custom rules on the local machine.

To turn off the FW_LO_NOTRACK option, edit the /etc/sysconfig/SuSEfirewall2 file and set FW_LO_NOTRACK="no". Save the file and restart the firewall or reboot.

For SUSE Linux Enterprise Server 12 SP3, the default value for FW_LO_NOTRACK was changed to no.

For Red Hat Enterprise Linux (RHEL) 8, if firewalld is running and FirewallBackend=nftables is set in /etc/firewalld/firewalld.conf, change this to FirewallBackend=iptables, or you can explicitly run the following commands to allow traffic to enter the default bridge (docker0) network:

firewall-cmd --permanent --zone=trusted --add-interface=docker0
firewall-cmd --reload

Time synchronization¶

In distributed systems like UCP, time synchronization is critical to ensure proper operation. As a best practice to ensure consistency between the engines in a UCP cluster, all engines should regularly synchronize time with a Network Time Protocol (NTP) server. If a host node’s clock is skewed, unexpected behavior may cause poor performance or even failures.

Load balancing strategy¶

UCP doesn’t include a load balancer. You can configure your own load balancer to balance user requests across all manager nodes.

If you plan to use a load balancer, you need to decide whether you’ll add the nodes to the load balancer using their IP addresses or their FQDNs. Whichever you choose, be consistent across nodes. When this is decided, take note of all IPs or FQDNs before starting the installation.

Load balancing UCP and DTR¶

By default, UCP and DTR both use port 443. If you plan on deploying UCP and DTR, your load balancer needs to distinguish traffic between the two by IP address or port number.

• If you want to configure your load balancer to listen on port 443:
  • Use one load balancer for UCP and another for DTR.
  • Use the same load balancer with multiple virtual IPs.
• Configure your load balancer to expose UCP or DTR on a port other than 443.

If you want to install UCP in a high-availability configuration that uses a load balancer in front of your UCP controllers, include the appropriate IP address and FQDN of the load balancer’s VIP by using one or more --san flags in the UCP install command or when you’re asked for additional SANs in interactive mode.

Use an external Certificate Authority¶

You can customize UCP to use certificates signed by an external Certificate Authority. When using your own certificates, you need to have a certificate bundle that has:

• A ca.pem file with the root CA public certificate,
• A cert.pem file with the server certificate and any intermediate CA public certificates. This certificate should also have SANs for all addresses used to reach the UCP manager,
• A key.pem file with server private key.

You can have a certificate for each manager, with a common SAN. For example, on a three-node cluster, you can have:

• node1.company.example.org with SAN ucp.company.org
• node2.company.example.org with SAN ucp.company.org
• node3.company.example.org with SAN ucp.company.org

You can also install UCP with a single externally-signed certificate for all managers, rather than one for each manager node. In this case, the certificate files are copied automatically to any new manager nodes joining the cluster or being promoted to a manager role.

Step 1: Validate the system requirements¶

The first step to installing UCP is ensuring that your infrastructure has all of the requirements UCP needs to run. Also, you need to ensure that all nodes, physical and virtual, are running the same version of Docker Enterprise.

Step 2: Install Docker Enterprise on all nodes¶

UCP is a containerized application that requires the commercially supported Docker Engine to run.

Install Docker Enterprise on each host that you plan to manage with UCP. View the supported platforms and click on your platform to get platform-specific instructions for installing Docker Enterprise.

Make sure you install the same Docker Enterprise version on all the nodes. Also, if you’re creating virtual machine templates with Docker Enterprise already installed, make sure the /etc/docker/key.json file is not included in the virtual machine image. When provisioning the virtual machine, restart the Docker daemon to generate a new /etc/docker/key.json file.

Step 3: Customize named volumes¶

Skip this step if you want to use the defaults provided by UCP.

UCP uses named volumes to persist data. If you want to customize the drivers used to manage these volumes, you can create the volumes before installing UCP. When you install UCP, the installer will notice that the volumes already exist, and it will start using them.

If these volumes don’t exist, they’ll be automatically created when installing UCP.

Step 4: Install UCP¶

To install UCP, you use the docker/ucp image, which has commands to install and manage UCP.

Make sure you follow the UCP System requirements for opening networking ports. Ensure that your hardware or software firewalls are open appropriately or disabled.

1. Use ssh to log in to the host where you want to install UCP.

2. Run the following command:

   # Pull the latest version of UCP
   docker image pull docker/ucp:3.2.5
   
   # Install UCP
   docker container run --rm -it --name ucp \
     -v /var/run/docker.sock:/var/run/docker.sock \
     docker/ucp:3.2.5 install \
     --host-address <node-ip-address> \
     --interactive
   

   This runs the install command in interactive mode, so that you’re prompted for any necessary configuration values. To find what other options are available in the install command, including how to install UCP on a system with SELinux enabled, check the reference documentation.

Step 5: License your installation¶

Now that UCP is installed, you need to license it. To use UCP, you are required to have a Docker Enterprise subscription, or you can test the platform with a free trial license.

1. Go to Docker Hub to get a free trial license.

2. In your browser, navigate to the UCP web UI, log in with your administrator credentials and upload your license. Navigate to the Admin Settings page and in the left pane, click License.

   

3. Click Upload License and navigate to your license (.lic) file. When you’re finished selecting the license, UCP updates with the new settings.

Step 6: Join manager nodes¶

To make your Docker swarm and UCP fault-tolerant and highly available, you can join more manager nodes to it. Manager nodes are the nodes in the swarm that perform the orchestration and swarm management tasks, and dispatch tasks for worker nodes to execute.

To join manager nodes to the swarm,

1. In the UCP web UI, navigate to the Nodes page, and click the Add Node button to add a new node.

   

2. In the Add Node page, check Add node as a manager to turn this node into a manager and replicate UCP for high-availability.

3. If you want to customize the network and port where the new node listens for swarm management traffic, click Use a custom listen address. Enter the IP address and port for the node to listen for inbound cluster management traffic. The format is interface:port or ip:port. The default is 0.0.0.0:2377.

4. If you want to customize the network and port that the new node advertises to other swarm members for API access, click Use a custom advertise address and enter the IP address and port. By default, this is also the outbound address used by the new node to contact UCP. The joining node should be able to contact itself at this address. The format is interface:port or ip:port.

5. Click the copy icon to copy the docker swarm join command that nodes use to join the swarm.

   

6. For each manager node that you want to join to the swarm, log in using ssh and run the join command that you copied. After the join command completes, the node appears on the Nodes page in the UCP web UI.

Step 7: Join worker nodes¶

To add more computational resources to your swarm, you can join worker nodes. These nodes execute tasks assigned to them by the manager nodes. Follow the same steps as before, but don’t check the Add node as a manager option.

Versions available¶

Use a computer with internet access to download the UCP package from the following links.

Download the offline package¶

You can also use these links to get the UCP package from the command line:

$ wget <ucp-package-url> -O ucp.tar.gz

Now that you have the package in your local machine, you can transfer it to the machines where you want to install UCP.

For each machine that you want to manage with UCP:

1. Copy the UCP package to the machine.

   $ scp ucp.tar.gz <user>@<host>
   

2. Use ssh to log in to the hosts where you transferred the package.

3. Load the UCP images.

   Once the package is transferred to the hosts, you can use the docker load command, to load the Docker images from the tar archive:

   $ docker load -i ucp.tar.gz
   

Follow the same steps for the DTR binaries.

Install UCP¶

Now that the offline hosts have all the images needed to install UCP, you can install UCP on one of the manager nodes.

Install UCP on AWS¶

Universal Control Plane (UCP) can be installed on top of AWS without any customisation following the UCP install documentation. Therefore this document is optional, however if you are deploying Kubernetes workloads with UCP and want to leverage the AWS kubernetes cloud provider, which provides dynamic volume and loadbalancer provisioning then you should follow this guide. This guide is not required if you are only deploying swarm workloads.

The requirements for installing UCP on AWS are included in the following sections:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [ "ec2:*" ],
      "Resource": [ "*" ]
    },
    {
      "Effect": "Allow",
      "Action": [ "elasticloadbalancing:*" ],
      "Resource": [ "*" ]
    },
    {
      "Effect": "Allow",
      "Action": [ "route53:*" ],
      "Resource": [ "*" ]
    },
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [ "arn:aws:s3:::kubernetes-*" ]
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [ "arn:aws:s3:::kubernetes-*" ]
    },
    {
      "Effect": "Allow",
      "Action": "ec2:Describe*",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "ec2:AttachVolume",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "ec2:DetachVolume",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [ "route53:*" ],
      "Resource": [ "*" ]
    }
}

$ docker container run --rm -it \
  --name ucp \
  --volume /var/run/docker.sock:/var/run/docker.sock \
  docker/ucp:3.2.5 install \
  --host-address <ucp-ip> \
  --cloud-provider aws \
  --interactive

Install UCP on Azure¶

Universal Control Plane (UCP) closely integrates with Microsoft Azure for its Kubernetes Networking and Persistent Storage feature set. UCP deploys the Calico CNI provider. In Azure, the Calico CNI leverages the Azure networking infrastructure for data path networking and the Azure IPAM for IP address management. There are infrastructure prerequisites required prior to UCP installation for the Calico / Azure integration.

{
    "cloud":"AzurePublicCloud",
    "tenantId": "***",
    "subscriptionId": "***",
    "aadClientId": "***",
    "aadClientSecret": "***",
    "resourceGroup": "***",
    "location": "***",
    "subnetName": "***",
    "securityGroupName": "***",
    "vnetName": "***",
    "useInstanceMetadata": true
}

"networkProfile": {
  "networkInterfaceConfigurations": [
    {
      "name": "[variables('nicName')]",
      "properties": {
        "ipConfigurations": [
          {
            "name": "[variables('ipConfigName1')]",
            "properties": {
              "primary": "true",
              "subnet": {
                "id": "[concat('/subscriptions/', subscription().subscriptionId,'/resourceGroups/', resourceGroup().name, '/providers/Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'), '/subnets/', variables('subnetName'))]"
              },
              "loadBalancerBackendAddressPools": [
                {
                  "id": "[concat('/subscriptions/', subscription().subscriptionId,'/resourceGroups/', resourceGroup().name, '/providers/Microsoft.Network/loadBalancers/', variables('loadBalancerName'), '/backendAddressPools/', variables('bePoolName'))]"
                }
              ],
              "loadBalancerInboundNatPools": [
                {
                  "id": "[concat('/subscriptions/', subscription().subscriptionId,'/resourceGroups/', resourceGroup().name, '/providers/Microsoft.Network/loadBalancers/', variables('loadBalancerName'), '/inboundNatPools/', variables('natPoolName'))]"
                }
              ]
            }
          },
          {
            "name": "[variables('ipConfigName2')]",
            "properties": {
              "subnet": {
                "id": "[concat('/subscriptions/', subscription().subscriptionId,'/resourceGroups/', resourceGroup().name, '/providers/Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'), '/subnets/', variables('subnetName'))]"
              }
            }
          }
          .
          .
          .
          {
            "name": "[variables('ipConfigName32')]",
            "properties": {
              "subnet": {
                "id": "[concat('/subscriptions/', subscription().subscriptionId,'/resourceGroups/', resourceGroup().name, '/providers/Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'), '/subnets/', variables('subnetName'))]"
              }
            }
          }
        ],
        "primary": "true"
      }
    }
  ]
}

docker container run --rm -it \
  --name ucp \
  --volume /var/run/docker.sock:/var/run/docker.sock \
  docker/ucp:3.2.5 install \
  --host-address <ucp-ip> \
  --pod-cidr <ip-address-range> \
  --cloud-provider Azure \
  --interactive

Plan the upgrade¶

As part of the upgrade process, you’ll upgrade the Docker EE Engine installed on each node of the cluster to version 17.06.2-ee-8 or higher. You should plan for the upgrade to take place outside of business hours, to ensure there’s minimal impact to your users.

Also, don’t make changes to UCP configurations while you’re upgrading it. This can lead to misconfigurations that are difficult to troubleshoot.

Ensure that your cluster nodes meet the minimum requirements for memory and disk space. In particular, manager nodes must have at least 8GB of memory.

Ensure that your cluster nodes meet the minimum requirements for port openings. The ports in use are documented in the UCP system requirements.

Back up your cluster¶

Before starting an upgrade, make sure that your cluster is healthy. If a problem occurs, this makes it easier to find and troubleshoot it.

Create a backup of your cluster. This allows you to recover if something goes wrong during the upgrade process.

Upgrade Docker Engine¶

For each node that is part of your cluster, upgrade the Docker Engine installed on that node to Docker Engine version 19.03 or higher. Be sure to install the Docker Enterprise Edition.

Starting with the manager nodes, and then worker nodes:

1. Log into the node using ssh.
2. Upgrade the Docker Engine to version 18.09.0 or higher.
3. Make sure the node is healthy.

Upgrade UCP¶

You can upgrade UCP from the web or the command line interface.

Use the web interface to perform an upgrade¶

When an upgrade is available for a UCP installation, a banner appears.

Clicking this message takes an admin user directly to the upgrade process. It can be found under the Upgrade tab of the Admin Settings section.

In the Available Versions dropdown, select the version you want to update to and click Upgrade UCP.

During the upgrade, the web interface will be unavailable, and you should wait until completion before continuing to interact with it. When the upgrade completes, you’ll see a notification that a newer version of the web interface is available and a browser refresh is required to see it.

# Get the latest version of UCP
docker image pull docker/ucp:3.1.8

docker container run --rm -it \
  --name ucp \
  -v /var/run/docker.sock:/var/run/docker.sock \
  docker/ucp:3.2.5 \
  upgrade --interactive

Download the offline package¶

You can also use these links to get the UCP package from the command line:

$ wget <ucp-package-url> -O ucp.tar.gz

Now that you have the package in your local machine, you can transfer it to the machines where you want to upgrade UCP.

For each machine that you want to manage with UCP:

1. Copy the offline package to the machine.

   $ scp ucp.tar.gz <user>@<host>
   

2. Use ssh to log in to the hosts where you transferred the package.

3. Load the UCP images.

   Once the package is transferred to the hosts, you can use the docker load command, to load the Docker images from the tar archive:

   $ docker load -i ucp.tar.gz
   

Upgrade UCP¶

Now that the offline hosts have all the images needed to upgrade UCP, you can upgrade UCP.

Swarm mode CA¶

After uninstalling UCP, the nodes in your cluster will still be in swarm mode, but you can’t join new nodes until you reinstall UCP, because swarm mode relies on UCP to provide the CA certificates that allow nodes in the cluster to identify one another. Also, since swarm mode is no longer controlling its own certificates, if the certificates expire after you uninstall UCP, the nodes in the swarm won’t be able to communicate at all. To fix this, either reinstall UCP before the certificates expire or disable swarm mode by running docker swarm leave --force on every node.

Restore IP tables¶

When you install UCP, the Calico network plugin changes the host’s IP tables. When you uninstall UCP, the IP tables aren’t reverted to their previous state. After you uninstall UCP, restart the node to restore its IP tables.