Perform the upgrade

This topic describes the following three different methods of upgrading MKE:

Note

To upgrade MKE on machines that are not connected to the Internet, refer to Install MKE offline to learn how to download the MKE package for offline installation.

In all three methods, manager nodes are automatically upgraded in place. You cannot control the order of manager node upgrades. For each worker node that requires an upgrade, you can upgrade that node in place or you can replace the node with a new worker node. The type of upgrade you perform depends on what is needed for each node.

Consult the following table to determine which method is right for you:

Upgrade method

Description

Automated in-place cluster upgrade

Performed on any manager node. This method automatically upgrades the entire cluster.

Phased in-place cluster upgrade

Automatically upgrades manager nodes and allows you to control the upgrade order of worker nodes. This type of upgrade is more advanced than the automated in-place cluster upgrade.

Replace existing worker nodes using blue-green deployment

This type of upgrade allows you to stand up a new cluster in parallel to the current one and switch over when the upgrade is complete. It requires that you join new worker nodes, schedule workloads to run on them, pause, drain, and remove old worker nodes in batches (rather than one at a time), and shut down servers to remove worker nodes. This is the most advanced upgrade method.

Automated in-place cluster upgrade method:

This is the standard method of upgrading MKE. It updates all MKE components on all nodes within the MKE cluster one-by-one until the upgrade is complete, and is thus not ideal for those needing to upgrade their worker nodes in a particular order.

  1. Verify that all MCR instances have been upgraded to the corresponding new version.

  2. SSH into one MKE manager node and run the following command (do not run this command on a workstation with a client bundle):

    docker container run --rm -it \
    --name ucp \
    --volume /var/run/docker.sock:/var/run/docker.sock \
    mirantis/ucp:3.3.16 \
    upgrade \
    --interactive
    

    The upgrade command will print messages as it automatically upgrades MKE on all nodes in the cluster.

Phased in-place cluster upgrade

This method allows granular control of the MKE upgrade process by first upgrading a manager node and then allowing you to upgrade worker nodes manually in the order that you select. This allows you to migrate workloads and control traffic while upgrading. You can temporarily run MKE worker nodes with different versions of MKE and MCR.

This method allows you to handle failover by adding additional worker node capacity during an upgrade. You can add worker nodes to a partially-upgraded cluster, migrate workloads, and finish upgrading the remaining worker nodes.

  1. Verify that all MCR instances have been upgraded to the corresponding new version.

  2. SSH into one MKE manager node and run the following command (do not run this command on a workstation with a client bundle):

    docker container run --rm -it \
    --name ucp \
    --volume /var/run/docker.sock:/var/run/docker.sock \
    mirantis/ucp:3.3.16 \
    upgrade \
    --manual-worker-upgrade \
    --interactive
    

    The --manual-worker-upgrade flag allows MKE to upgrade only the manager nodes. It adds an upgrade-hold label to all worker nodes, which prevents MKE from upgrading each worker node until you remove the label.

  3. Optional. Join additional worker nodes to your cluster:

    docker swarm join --token SWMTKN-<swarm-token> <manager-ip>:2377
    

    For more information, refer to Join Linux nodes.

    Note

    New worker nodes will already have the newer version of MCR and MKE installed when they join the cluster.

  4. Remove the upgrade-hold label from each worker node to upgrade:

    docker node update --label-rm com.docker.ucp.upgrade-hold \
    <node-name-or-id>
    

Replace existing worker nodes using blue-green deployment

This method creates a parallel environment for a new deployment, which reduces downtime, upgrades worker nodes without disrupting workloads, and allows you to migrate traffic to the new environment with worker node rollback capability.

Note

You do not have to replace all worker nodes in the cluster at one time, but can instead replace them in groups.

  1. Verify that all MCR instances have been upgraded to the corresponding new version.

  2. SSH into one MKE manager node and run the following command (do not run this command on a workstation with a client bundle):

    docker container run --rm -it \
    --name ucp \
    --volume /var/run/docker.sock:/var/run/docker.sock \
    mirantis/ucp:3.3.16 \
    upgrade \
    --manual-worker-upgrade \
    --interactive
    

    The --manual-worker-upgrade flag allows MKE to upgrade only the manager nodes. It adds an upgrade-hold label to all worker nodes, which prevents MKE from upgrading each worker node until the label is removed.

  3. Join additional worker nodes to your cluster:

    docker swarm join --token SWMTKN-<swarm-token> <manager-ip>:2377
    

    For more information, refer to Join Linux nodes.

    Note

    New worker nodes will already have the newer version of MCR and MKE installed when they join the cluster.

  4. Join MCR to the cluster:

    docker swarm join --token SWMTKN-<your-token> <manager-ip>:2377
    
  5. Pause all existing worker nodes to ensure that MKE does not deploy new workloads on existing nodes:

    docker node update --availability pause <node-name>
    
  6. Drain the paused nodes in preparation for migrating your workloads:

    docker node update --availability drain <node-name>
    

    Note

    MKE automatically reschedules workloads onto new nodes while existing nodes are paused.

  7. Remove each fully-drained node:

    docker swarm leave <node-name>
    
  8. Remove each manager node after its worker nodes become unresponsive:

    docker node rm <node-name>
    
  9. From any manager node, remove old MKE agents after the upgrade is complete, including 390x and Windows agents carried over from the previous install:

    docker service rm ucp-agent
    docker service rm ucp-agent-win
    docker service rm ucp-agent-s390x