Automatically update or upgrade Kubernetes¶

This section describes how to update or upgrade your Kubernetes cluster including Calico and etcd [0] using the Jenkins Deploy - update Kubernetes cluster pipeline.

To update or upgrade Kubernetes using the Jenkins pipeline:

1. Log in to the Jenkins web UI.

2. Open the Deploy - update Kubernetes cluster pipeline.

3. Specify the following parameters:

   Parameter	Description and values
   ARTIFACTORY_URL	Optional. Required for conformance tests only. The artifactory URL where Docker images are located. Automatically taken from Reclass.
   CALICO_UPGRADE_VERSION	Define the version of the calico-upgrade utility to use during the Calico upgrade. This option is only relevant if UPGRADE_CALICO_V2_TO_V3 is selected.
   CMP_TARGET	Add the target Kubernetes cmp nodes. For example, 'cmp* and I@kubernetes:pool'.
   CONFORMANCE_RUN_AFTER	Optional. Select to run the Kubernetes conformance tests after you upgrade a Kubernetes cluster.
   CONFORMANCE_RUN_BEFORE	Optional. Select to run the Kubernetes conformance tests before you upgrade a Kubernetes cluster.
   CTL_TARGET	Add the target Kubernetes ctl nodes. For example, I@kubernetes:master.
   KUBERNETES_CALICO_CALICOCTL_SOURCE KUBERNETES_CALICO_CALICOCTL_SOURCE_HASH KUBERNETES_CALICO_CNI_SOURCE_HASH KUBERNETES_CALICO_BIRDCL_SOURCE KUBERNETES_CALICO_BIRDCL_SOURCE_HASH KUBERNETES_CALICO_CNI_IPAM_SOURCE KUBERNETES_CALICO_CNI_IPAM_SOURCE_HASH	Leave these fields empty since you have already updated your MCP cluster to a newer Build ID as described in Upgrade DriveTrain to a newer release version.
   KUBERNETES_CALICO_CNI_SOURCE	Leave this field empty since you have already updated your MCP cluster to a newer Build ID as described in Upgrade DriveTrain to a newer release version. For testing purposes, you can add a versioned calico/cni image to use in your deployments.
   KUBERNETES_CALICO_IMAGE	Leave this field empty since you have already updated your MCP cluster to a newer Build ID as described in Upgrade DriveTrain to a newer release version. For testing purposes, you can add a versioned calico/node image to use in your deployments.
   KUBERNETES_CALICO_KUBE_CONTROLLERS_IMAGE	Leave this field empty since you have already updated your MCP cluster to a newer Build ID as described in Upgrade DriveTrain to a newer release version. For testing purposes, you can add a versioned calico/kube-controllers image to use in your deployments.
   KUBERNETES_ETCD_SOURCE [0]	Leave this field empty since you have already updated your MCP cluster to a newer Build ID as described in Upgrade DriveTrain to a newer release version. For testing purposes, you can add a versioned binary of the etcd server to use in your deployment.
   KUBERNETES_ETCD_SOURCE_HASH [0]	Leave this field empty since you have already updated your MCP cluster to a newer Build ID as described in Upgrade DriveTrain to a newer release version. For testing purposes, you can add the checksum of the versioned etcd binary set in the KUBERNETES_ETCD_SOURCE field.
   KUBERNETES_HYPERKUBE_SOURCE	Leave this field empty since you have already updated your MCP cluster to a newer Build ID as described in Upgrade DriveTrain to a newer release version. For testing purposes, you can add a versioned image to update the Kubernetes Master nodes from. Also, verify that the - cluster.overrides class exists your cluster model. Otherwise, the KUBERNETES_HYPERKUBE_SOURCE, KUBERNETES_PAUSE_IMAGE, KUBERNETES_CALICO_IMAGE, KUBERNETES_CALICO_CALICOCTL_SOURCE, KUBERNETES_CALICO_CNI_SOURCE, and KUBERNETES_CALICO_KUBE_CONTROLLERS_IMAGE variables will not take effect.
   KUBERNETES_HYPERKUBE_SOURCE_HASH	Leave this field empty since you have already updated your MCP cluster to a newer Build ID as described in Upgrade DriveTrain to a newer release version.
   KUBERNETES_PAUSE_IMAGE	Leave this field empty since you have already updated your MCP cluster to a newer Build ID as described in Upgrade DriveTrain to a newer release version. For testing purposes, you can add a versioned image to use in your deployments.
   PER_NODE	Select to update or upgrade the target Kubernetes nodes one by one. The option is required for the draining and cordoning functionality. Recommended.
   SALT_MASTER_CREDENTIALS	The Salt Master credentials to use for connection, defaults to salt.
   SALT_MASTER_URL	The Salt Master node host URL with the salt-api port, defaults to the jenkins_salt_api_url parameter. For example, http://172.18.170.27:6969.
   SIMPLE_UPGRADE	Select if you do not need to drain and cordon the nodes during the cluster upgrade. Not recommended.
   TARGET_UPDATES	Add the comma-separated list of the Kubernetes nodes to update. Valid values are ctl, cmp. To update only the Kubernetes Nodes, for example, define cmp only.
   TEST_K8S_API_SERVER	Optional. Required for conformance tests only. The IP address of a local Kubernetes API server for conformance tests.
   UPGRADE_CALICO_V2_TO_V3	Select to upgrade Calico from version 2.6.x to the latest supported version (3.x). This option is required only for upgrade of Calico from version 2.6.x. To update the minor Calico version (for example, from 3.1.x to 3.3.x), do not select this option, since the regular Kubernetes update procedure already includes the update of Calico minor version. Caution The Calico upgrade process implies the Kubernetes services downtime for workloads operations, for example, workloads spawning and removing. The downtime is caused by the necessity of the etcd schema migration where the Calico endpoints data and other Calico configuration data is stored.

4. Click Deploy. To monitor the deployment process, follow the instruction in MCP Deployment Guide: View the deployment details.

5. ^{Obsolete since 2019.2.3} If you do not have any third-party Docker workloads that run outside the MCP Kubernetes cluster, stop and disable Docker on reboot. Run the following commands on any Kubernetes Master node:

   systemctl stop docker
   systemctl disable docker
   

   Note

   If your MCP cluster version is 2019.2.3 or later, skip this step since Docker is removed during the upgrade.

The Deploy - update Kubernetes cluster pipeline workflow:

1. Add the hyperkube images defined in the KUBERNETES_HYPERKUBE_SOURCE, KUBERNETES_HYPERKUBE_SOURCE_HASH, and KUBERNETES_PAUSE_IMAGE pipeline fields or defined on the reclass-system level of the Reclass model for the target Kubernetes ctl and cmp nodes during the MCP release version upgrade.
2. Add the calico images defined in the KUBERNETES_CALICO_IMAGE, KUBERNETES_CALICO_CALICOCTL_SOURCE, KUBERNETES_CALICO_CALICOCTL_SOURCE_HASH, KUBERNETES_CALICO_CNI_SOURCE, KUBERNETES_CALICO_CNI_SOURCE_HASH, KUBERNETES_CALICO_BIRDCL_SOURCE, KUBERNETES_CALICO_BIRDCL_SOURCE_HASH, KUBERNETES_CALICO_CNI_IPAM_SOURCE, KUBERNETES_CALICO_CNI_IPAM_SOURCE_HASH, KUBERNETES_CALICO_KUBE_CONTROLLERS_IMAGE pipeline job parameters or defined on the reclass-system level of the Reclass model for the target Kubernetes ctl and cmp nodes during the MCP release version upgrade.
3. If the UPGRADE_CALICO_V2_TO_V3 option is selected, perform the Calico upgrade:
   1. Download the calico-upgrade utility according to the CALICO_UPGRADE_VERSION setting.
   2. Verify that the Calico upgrade is possible (verify the current Calico version and perform a dry run of the data upgrade).
   3. Verify the Calico policy setting.
   4. Perform the Calico data upgrade and lock Calico.
   5. Update the Calico configuration and restart corresponding services. Calico is not operating during this step, so a cluster experiences the workloads operations downtime.
   6. Unlock Calico.
4. Upgrade etcd on the Kubernetes target ctl nodes one by one.
5. For the first Kubernetes target ctl node:
   1. Cordon the node.
   2. Drain the node.
   3. Regenerate SSL certificates for the node services.
   4. Start the containerd service.
   5. Upgrade the hyperkube image.
   6. Restart the node services (api-server, api-scheduler, controller-manager, kube-proxy).
   7. Restart the kubelet service.
   8. If any DaemonSet is deployed on a cluster:
      1. Force remove the DaemonSet pod.
      2. Reboot the node.
   9. Uncordon the node.
6. Complete the previous step on the remaining target ctl nodes one by one.
7. Upgrade add-ons if any on the target ctl nodes.
8. For the first Kubernetes target cmp node:
   1. Cordon the node.
   2. Drain the node.
   3. Regenerate SSL certificates for the node services.
   4. Start the containerd service.
   5. Upgrade the hyperkube image.
   6. Restart the kubelet service.
   7. If any DaemonSet is deployed on a cluster:
      1. Force remove the DaemonSet pod.
      2. Reboot the node.
   8. Uncordon the node.
9. Complete the previous step on the remaining target cmp nodes one by one.
10. Verify the Calico cluster version and integrity.