Perform the upgrade

Note

• If the MKE Interlock configuration is customized, the Interlock component is managed by the user and thus cannot be upgraded using the upgrade command. In such cases, Interlock must be manually upgraded using Docker, as follows:

  docker service update --image mirantis/ucp-interlock:<upgrade-target-mke-version>ucp-interlock
  docker service update --image mirantis/ucp-interlock-extension:<upgrade-target-mke-version> ucp-interlock-extension
  docker service update --image mirantis/ucp-interlock-proxy:<upgrade-target-mke-version> ucp-interlock-proxy \
  --debug
  

• To upgrade MKE on machines that are not connected to the Internet, refer to Install MKE offline to learn how to download the MKE package for offline installation.

• To manually interrupt the upgrade process, enter Control-C on the terminal upon which you have initiated the upgrade bootstrapper. Doing so will trigger an automatic rollback to the previous MKE version.

• If no upgrade progress is made within 20 minutes, MKE will initiate a rollback to the original version.

The upgrade methods for MKE are:

• Automated in-place cluster upgrade

• Phased in-place cluster upgrade

• Replace existing worker nodes using blue-green deployment

With all of these upgrade methods, manager nodes are automatically upgraded in place. You cannot control the order of manager node upgrades. For each worker node that requires an upgrade, you can upgrade that node in place or you can replace the node with a new worker node. The type of upgrade you perform depends on what is needed for each node.

Automated rollbacks are only supported when MKE is in control of the upgrade process, which is while the upgrade containers are running. As such, the feature scope is limited in terms of any failures encountered during Phased in-place cluster upgrade and Replace existing worker nodes using blue-green deployment upgrade methods.

Upgrade method	Description	Automated rollback support
Automated in-place cluster upgrade	Performed on any manager node. This method automatically upgrades the entire cluster.	Yes.
Phased in-place cluster upgrade	Automatically upgrades manager nodes and allows you to control the upgrade order of worker nodes. This type of upgrade is more advanced than the automated in-place cluster upgrade.	Only if the failure occurs before or during manager node upgrade.
Replace existing worker nodes using blue-green deployment	This type of upgrade allows you to stand up a new cluster in parallel to the current one and switch over when the upgrade is complete. It requires that you join new worker nodes, schedule workloads to run on them, pause, drain, and remove old worker nodes in batches (rather than one at a time), and shut down servers to remove worker nodes. This is the most advanced upgrade method.	Only if the failure occurs before or during manager node upgrade.

Automated in-place cluster upgrade method

Automated in-place cluster upgrade is the standard method for upgrading MKE. It updates all MKE components on all nodes within the MKE cluster one-by-one until the upgrade is complete, and thus it is not ideal for those who need to upgrade their worker nodes in a particular order.

1. Verify that all MCR instances have been upgraded to the corresponding new version.

2. SSH into one MKE manager node and run the following command (do not run this command on a workstation with a client bundle):

   docker container run --rm -it \
   --name ucp \
   --volume /var/run/docker.sock:/var/run/docker.sock \
   mirantis/ucp:3.7.8 \
   upgrade \
   --interactive \
   --debug
   

   The upgrade command will print messages as it automatically upgrades MKE on all nodes in the cluster.

Phased in-place cluster upgrade

The Phased in-place cluster upgrade method allows for granular control of the MKE upgrade process by first upgrading a manager node and thereafter allowing you to upgrade worker nodes manually in your preferred order. This allows you to migrate workloads and control traffic while upgrading. You can temporarily run MKE worker nodes with different versions of MKE and MCR.

This method allows you to handle failover by adding additional worker node capacity during an upgrade. You can add worker nodes to a partially-upgraded cluster, migrate workloads, and finish upgrading the remaining worker nodes.

1. Verify that all MCR instances have been upgraded to the corresponding new version.

2. SSH into one MKE manager node and run the following command (do not run this command on a workstation with a client bundle):

   docker container run --rm -it \
   --name ucp \
   --volume /var/run/docker.sock:/var/run/docker.sock \
   mirantis/ucp:3.7.8 \
   upgrade \
   --manual-worker-upgrade \
   --interactive \
   --debug
   

   The --manual-worker-upgrade flag allows MKE to upgrade only the manager nodes. It adds an upgrade-hold label to all worker nodes, which prevents MKE from upgrading each worker node until you remove the label.

3. Optional. Join additional worker nodes to your cluster:

   docker swarm join --token SWMTKN-<swarm-token> <manager-ip>:2377
   

   For more information, refer to Join Linux nodes.

   Note

   New worker nodes will already have the newer version of MCR and MKE installed when they join the cluster.

4. Remove the upgrade-hold label from each worker node to upgrade:

   docker node update --label-rm com.docker.ucp.upgrade-hold \
   <node-name-or-id>
   

Replace existing worker nodes using blue-green deployment

Th Replace existing worker nodes using blue-green deployment upgrade method creates a parallel environment for a new deployment, which reduces downtime, upgrades worker nodes without disrupting workloads, and allows you to migrate traffic to the new environment with worker node rollback capability.

Note

You do not have to replace all worker nodes in the cluster at one time, but can instead replace them in groups.

1. Verify that all MCR instances have been upgraded to the corresponding new version.

2. SSH into one MKE manager node and run the following command (do not run this command on a workstation with a client bundle):

   docker container run --rm -it \
   --name ucp \
   --volume /var/run/docker.sock:/var/run/docker.sock \
   mirantis/ucp:3.7.8 \
   upgrade \
   --manual-worker-upgrade \
   --interactive \
   --debug
   

   The --manual-worker-upgrade flag allows MKE to upgrade only the manager nodes. It adds an upgrade-hold label to all worker nodes, which prevents MKE from upgrading each worker node until the label is removed.

3. Join additional worker nodes to your cluster:

   docker swarm join --token SWMTKN-<swarm-token> <manager-ip>:2377
   

   For more information, refer to Join Linux nodes.

   Note

   New worker nodes will already have the newer version of MCR and MKE installed when they join the cluster.

4. Join MCR to the cluster:

   docker swarm join --token SWMTKN-<your-token> <manager-ip>:2377
   

5. Pause all existing worker nodes to ensure that MKE does not deploy new workloads on existing nodes:

   docker node update --availability pause <node-name>
   

6. Drain the paused nodes in preparation for migrating your workloads:

   docker node update --availability drain <node-name>
   

   Note

   MKE automatically reschedules workloads onto new nodes while existing nodes are paused.

7. Remove each fully-drained node:

   docker swarm leave <node-name>
   

8. Remove each manager node after its worker nodes become unresponsive:

   docker node rm <node-name>
   

9. From any manager node, remove old MKE agents after the upgrade is complete, including 390x and Windows agents carried over from the previous install:

   docker service rm ucp-agent
   docker service rm ucp-agent-win
   docker service rm ucp-agent-s390x