Create update groups for worker machines

The use of update groups provides enhanced control over update of worker machines by allowing granular concurrency settings for specific machine groups. This feature uses the UpdateGroup object to decouple the concurrency settings from the global cluster level, providing flexibility based on the workload characteristics of different machine sets.

The UpdateGroup objects are processed sequentially based on their indexes. Update groups with the same indexes are processed concurrently. The control update group is always processed first.

Note

The update order of a machine within the same group is determined by the upgrade index of a specific machine. For details, see Change the upgrade order of a machine.

The maxWorkerUpgradeCount parameter of the Cluster object is inherited by the default update group. Changing maxWorkerUpgradeCount leads to changing the concurrentUpdates parameter of the default update group.

Note

The maxWorkerUpgradeCount parameter of the Cluster object is deprecated and will be removed in one of the following releases. You can still use this parameter to change the concurrentUpdates value of the default update group. However, Mirantis recommends changing this value directly in the UpdateGroup object.

The concurrentUpdates parameter defines how many machines in a group can be processed concurrently by lcm-controller. However, it does not guarantee that exactly this number of machines will be updated at the same time, as the actual concurrency also depends on the roles of each machine. For details, see Parallelizing node update operations.

Update group for controller nodes

TechPreview

The update group for controller nodes is automatically generated during initial cluster creation with the following settings:

name: <cluster-name>-control
index: 1
concurrentUpdates: 1
rebootIfUpdateRequires: false

Caution

During a distribution upgrade, machines are always rebooted, overriding rebootIfUpdateRequires: false.

All control plane machines are automatically assigned to the update group for controller nodes with no possibility to change it.

Caution

The index and concurrentUpdates parameters of the update group for controller nodes are hardcoded and cannot be changed.

Example of the update group for controller nodes:

apiVersion: kaas.mirantis.com/v1alpha1
kind: UpdateGroup
metadata:
  name: example-cluster-control
  namespace: example-ns
  labels:
    cluster.sigs.k8s.io/cluster-name: example-cluster
spec:
  index: 1
  concurrentUpdates: 1
  rebootIfUpdateRequires: false

Default update group

The default update group is automatically created during initial cluster creation with the following settings:

name: <cluster-name>-default
index: 1
rebootIfUpdateRequires: false
concurrentUpdates: inherited from the maxWorkerUpgradeCount parameter set in the Cluster object

Caution

During a distribution upgrade, machines are always rebooted, overriding rebootIfUpdateRequires: false.

Example of the default update group:

apiVersion: kaas.mirantis.com/v1alpha1
kind: UpdateGroup
metadata:
  name: example-cluster-default
  namespace: example-ns
  labels:
    cluster.sigs.k8s.io/cluster-name: example-cluster
spec:
  index: 1
  concurrentUpdates: 1
  rebootIfUpdateRequires: false

If you require custom update settings for worker machines, create one or several custom UpdateGroup objects as described below.

Assign a machine to an update group using CLI

Note

All worker machines that are not assigned to any update group are automatically assigned to the default update group.

Create an UpdateGroup object with the required specification. For description of the object fields, see UpdateGroup resource.
Label the machines to associate them with the newly created UpdateGroup object:
```
kubectl label machine <machineName> kaas.mirantis.com/update-group=<UpdateGroupObjectName>
```
To change the update group of a machine, update the kaas.mirantis.com/update-group label of the machine with the new update group name. Removing this label from a machine automatically assigns such machine to the default update group.
Strongly recommended. Back up MKE as described in Mirantis Kubernetes Engine documentation: Back up MKE.

Since the procedure above modifies the cluster configuration, a fresh backup is required to restore the cluster in case further reconfigurations fail.
Important

Because the MKE restoration process is complicated, we strongly recommend contacting Mirantis support for assistance.

If you still decide to restore MKE from a backup on your own, you must scale down helm-controller on the cluster being restored if the MKE version of the affected cluster after the restore will differ from the MKE version in the ClusterRelease object that is set in MOSK Cluster objects in the management cluster:
- If you are restoring MKE on a management cluster: before starting the restore, scale down helm-controller on each affected MOSK cluster. This prevents unintended Ceph and OpenStack downgrades on MOSK clusters after the management cluster is restored.
- If you are restoring MKE on a MOSK cluster: immediately after the restore completes, scale down helm-controller. Because the restore rolls the cluster back to an older release, this prevents it from triggering a premature upgrade of Helm releases.

Note

After creation of a custom UpdateGroup object, if you plan to add a new machine that requires a non-default update group, manually add the corresponding label to the machine as described above. Otherwise, the default update group is applied to such machine.

Note

Before removing the UpdateGroup object, reassign all machines to another update group.