This section describes how to update or upgrade your Kubernetes cluster including Calico and etcd 0 using the Jenkins Deploy - update Kubernetes cluster pipeline.
To update or upgrade Kubernetes using the Jenkins pipeline:
Log in to the Jenkins web UI.
Open the Deploy - update Kubernetes cluster pipeline.
Specify the following parameters:
Parameter |
Description and values |
---|---|
ARTIFACTORY_URL |
Optional. Required for conformance tests only. The artifactory URL where Docker images are located. Automatically taken from Reclass. |
CALICO_UPGRADE_VERSION |
Define the version of the calico-upgrade utility to use during
the Calico upgrade. This option is only relevant if
|
CMP_TARGET |
Add the target Kubernetes |
CONFORMANCE_RUN_AFTER |
Optional. Select to run the Kubernetes conformance tests after you upgrade a Kubernetes cluster. |
CONFORMANCE_RUN_BEFORE |
Optional. Select to run the Kubernetes conformance tests before you upgrade a Kubernetes cluster. |
CTL_TARGET |
Add the target Kubernetes |
|
Leave these fields empty since you have already updated your MCP cluster to a newer Build ID as described in Upgrade DriveTrain to a newer release version. |
KUBERNETES_CALICO_CNI_SOURCE |
Leave this field empty since you have already updated
your MCP cluster to a newer Build ID as described in
Upgrade DriveTrain to a newer release version. For testing purposes, you can add
a versioned |
KUBERNETES_CALICO_IMAGE |
Leave this field empty since you have already updated
your MCP cluster to a newer Build ID as described in
Upgrade DriveTrain to a newer release version. For testing purposes, you can add
a versioned |
KUBERNETES_CALICO_KUBE_CONTROLLERS_IMAGE |
Leave this field empty since you have already updated
your MCP cluster to a newer Build ID as described in
Upgrade DriveTrain to a newer release version. For testing purposes, you can add
a versioned |
KUBERNETES_ETCD_SOURCE 0 |
Leave this field empty since you have already updated your MCP cluster to a newer Build ID as described in Upgrade DriveTrain to a newer release version. For testing purposes, you can add a versioned binary of the etcd server to use in your deployment. |
KUBERNETES_ETCD_SOURCE_HASH 0 |
Leave this field empty since you have already updated your MCP cluster to a newer Build ID as described in Upgrade DriveTrain to a newer release version. For testing purposes, you can add the checksum of the versioned etcd binary set in the KUBERNETES_ETCD_SOURCE field. |
KUBERNETES_HYPERKUBE_SOURCE |
Leave this field empty since you have already updated your MCP cluster to a newer Build ID as described in Upgrade DriveTrain to a newer release version. For testing purposes, you can add a versioned image to update
the Kubernetes Master nodes from. Also, verify that the
|
KUBERNETES_HYPERKUBE_SOURCE_HASH |
Leave this field empty since you have already updated your MCP cluster to a newer Build ID as described in Upgrade DriveTrain to a newer release version. |
KUBERNETES_PAUSE_IMAGE |
Leave this field empty since you have already updated your MCP cluster to a newer Build ID as described in Upgrade DriveTrain to a newer release version. For testing purposes, you can add a versioned image to use in your deployments. |
PER_NODE |
Select to update or upgrade the target Kubernetes nodes one by one. The option is required for the draining and cordoning functionality. Recommended. |
SALT_MASTER_CREDENTIALS |
The Salt Master credentials to use for connection, defaults to
|
SALT_MASTER_URL |
The Salt Master node host URL with the |
SIMPLE_UPGRADE |
Select if you do not need to drain and cordon the nodes during the cluster upgrade. Not recommended. |
TARGET_UPDATES |
Add the comma-separated list of the Kubernetes nodes to update.
Valid values are |
TEST_K8S_API_SERVER |
Optional. Required for conformance tests only. The IP address of a local Kubernetes API server for conformance tests. |
UPGRADE_CALICO_V2_TO_V3 |
Select to upgrade Calico from version 2.6.x to the latest supported version (3.x). This option is required only for upgrade of Calico from version 2.6.x. To update the minor Calico version (for example, from 3.1.x to 3.3.x), do not select this option, since the regular Kubernetes update procedure already includes the update of Calico minor version. Caution The Calico upgrade process implies the Kubernetes services downtime for workloads operations, for example, workloads spawning and removing. The downtime is caused by the necessity of the etcd schema migration where the Calico endpoints data and other Calico configuration data is stored. |
Click Deploy. To monitor the deployment process, follow the instruction in MCP Deployment Guide: View the deployment details.
Obsolete since 2019.2.3 If you do not have any third-party Docker workloads that run outside the MCP Kubernetes cluster, stop and disable Docker on reboot. Run the following commands on any Kubernetes Master node:
systemctl stop docker
systemctl disable docker
Note
If your MCP cluster version is 2019.2.3 or later, skip this step since Docker is removed during the upgrade.
The Deploy - update Kubernetes cluster pipeline workflow:
Note
While any Kubernetes node is being cordoned or drained, other cluster nodes run its services. Therefore, the workload is not affected.
Add the hyperkube
images defined in the
KUBERNETES_HYPERKUBE_SOURCE,
KUBERNETES_HYPERKUBE_SOURCE_HASH, and
KUBERNETES_PAUSE_IMAGE pipeline fields or defined
on the reclass-system
level of the Reclass model for
the target Kubernetes ctl
and cmp
nodes during
the MCP release version upgrade.
Add the calico
images defined in the
KUBERNETES_CALICO_IMAGE,
KUBERNETES_CALICO_CALICOCTL_SOURCE,
KUBERNETES_CALICO_CALICOCTL_SOURCE_HASH,
KUBERNETES_CALICO_CNI_SOURCE,
KUBERNETES_CALICO_CNI_SOURCE_HASH,
KUBERNETES_CALICO_BIRDCL_SOURCE,
KUBERNETES_CALICO_BIRDCL_SOURCE_HASH,
KUBERNETES_CALICO_CNI_IPAM_SOURCE,
KUBERNETES_CALICO_CNI_IPAM_SOURCE_HASH,
KUBERNETES_CALICO_KUBE_CONTROLLERS_IMAGE pipeline job parameters
or defined on the reclass-system
level of the Reclass model for
the target Kubernetes ctl
and cmp
nodes during
the MCP release version upgrade.
If the UPGRADE_CALICO_V2_TO_V3 option is selected, perform the Calico upgrade:
Download the calico-upgrade utility according to the CALICO_UPGRADE_VERSION setting.
Verify that the Calico upgrade is possible (verify the current Calico version and perform a dry run of the data upgrade).
Verify the Calico policy setting.
Perform the Calico data upgrade and lock Calico.
Update the Calico configuration and restart corresponding services. Calico is not operating during this step, so a cluster experiences the workloads operations downtime.
Unlock Calico.
Upgrade etcd on the Kubernetes target ctl
nodes one by one.
For the first Kubernetes target ctl
node:
Cordon the node.
Drain the node.
Regenerate SSL certificates for the node services.
Start the containerd
service.
Upgrade the hyperkube
image.
Restart the node services (api-server
, api-scheduler
,
controller-manager
, kube-proxy
).
Restart the kubelet
service.
If any DaemonSet is deployed on a cluster:
Force remove the DaemonSet pod.
Reboot the node.
Uncordon the node.
Complete the previous step on the remaining target ctl
nodes
one by one.
Upgrade add-ons if any on the target ctl
nodes.
For the first Kubernetes target cmp
node:
Cordon the node.
Drain the node.
Regenerate SSL certificates for the node services.
Start the containerd
service.
Upgrade the hyperkube
image.
Restart the kubelet
service.
If any DaemonSet is deployed on a cluster:
Force remove the DaemonSet pod.
Reboot the node.
Uncordon the node.
Complete the previous step on the remaining target cmp
nodes
one by one.
Verify the Calico cluster version and integrity.