Scale nodesNew

You can add and remove nodes using the mkectl node add and mkectl node remove commands, respectively, on a running MKE 4 cluster without having to perform a full cluster reconciliation. Each command runs a targeted sequence of steps, bootstrap or cordon/drain/reset, and updates the MkeConfig stored in the cluster to reflect the new node membership.

Both commands operate on either worker nodes or control plane (manager) nodes. The role that a command targets is selected with the --scope flag.

Node roles and the --scope flag

The --scope flag serves two purposes: it selects which entries of the host configuration file the command acts on, and it selects the execution path that mkectl runs.

--scope value	Targets	Notes
worker (default)	Entries in the hosts file with role: worker	Worker path. Fastest, least disruptive.
controller+worker	Entries in the hosts file with role: controller+worker	Control plane path. Adds etcd membership verification, pre- and post-operation cluster backups whenever etcd membership changes, and a cluster-wide operation lock.

A single command invocation acts on one role only. To scale both roles, run the command twice, once per scope.

Note

In MKE 4, a control plane node runs the k0s controller+worker role, which means that it is both a Kubernetes control plane node and a schedulable node. There is no separate controller-only role.

Limitations

• You must provision target nodes in advance. Cloud and virtual machine resources are not automatically created.
• Automatic recovery does not occur if an operation is interrupted before completion, and thus you may need to manually clean up the affected node.
• Windows worker nodes are not supported.
• The commands update MkeConfig only for the hosts explicitly listed in the operation. The hosts list is not otherwise reconciled against actual cluster membership. For more information, refer to MkeConfig update scope.
• Control plane specific:
  • Control plane nodes are joined and removed one at a time, regardless of the --concurrency setting.
  • A control plane operation takes a cluster-wide lock, and thus it cannot run concurrently with any other node operation. For more information, refer to Concurrent operations.
  • The etcd verification, member join, and member leave stages apply only to clusters that use the default stacked (internal) etcd. External etcd is not supported.
  • Removing a control plane node that would leave the cluster with an even number of etcd members requires the --force flag. For more information, refer to etcd quorum arithmetic.
  • Load balancer that routes traffic to control-planes should be updated manually when control plane nodes added/removed

Use the mkectl apply command to add Windows worker nodes

The mkectl node add command does not support Windows worker nodes. To add a Windows worker node, edit the spec.hosts section of the mke4.yaml configuration file and run the mkectl apply command. For all other node types, use the mkectl node add command. For more information, refer to Install Windows worker nodes.

The same rule applies when nodes rejoin the cluster after a restore: control plane nodes and Linux worker nodes rejoin through the mkectl node add command, and Windows worker nodes rejoin through the mkectl apply command. For more information, refer to Restore.

The mkectl apply command does not remove nodes from the cluster. Deleting a host from spec.hosts does not remove that node from the cluster.

Workload redistribution

When the mkectl node add command adds control plane nodes, MKE automatically redistributes its system components across all of the control plane nodes. It does this by performing a rolling restart of the MKE-managed Deployments and StatefulSets in the mke and k0rdent namespaces. These components restart briefly during the operation, which is normal behavior that requires no manual intervention.

• The redistribution runs only when the operation changes etcd membership. A re-run of the command against control plane nodes that have already joined the cluster does not restart the components.
• The redistribution does not run when you remove control plane nodes, as the drain moves the affected Pods to the remaining control plane nodes.
• The --dry-run output does not list the redistribution.

Note

If the mkectl node add operation joins the control plane nodes and then fails before the redistribution, a re-run of the command will not redistribute the components, as etcd membership no longer changes. In this case, once all of the control plane nodes report Ready, manually restart the components:

kubectl rollout restart deployment,statefulset -n mke
kubectl rollout restart deployment,statefulset -n k0rdent

Prerequisites and constraints

Version alignment

The mkectl binary version must match the version that was used to deploy your MKE 4 cluster. The k0s version that mkectl installs on new nodes must also match the k0s version running on existing cluster nodes.

Cluster state

MKE 4 tracks cluster-wide state to gate node scaling. Before executing, the mkectl node add and mkectl node remove commands both verify that the cluster is in a stable state and abort if it is not (stable in this context means that there is no other concurrent operations running with the cluster).

Operations that place the cluster in a non-stable state—such as upgrade, apply, and reset—will block node scaling until they complete. Cluster state is managed using lease objects that are stored in the cluster, in the mke namespace by default.

Etcd state

Etcd state should be operational (see below), for all scenarios: adding/removing managers/workers. Operational means: the readyz probe passes, a linearizable read succeeds, a leader is known, quorum holds, the backend database is not near its quota, and neither the NOSPACE nor the CORRUPT alarm is raised.

Node state (for workers only)

When workers scenarios executed, each node carries an operation signal that indicates whether it is being added, being removed, or remaining idle. Node state is managed through lease objects, which prevent concurrent conflicting operations on the same node; for example, attempting to simultaneously add and remove the same node.

Concurrent operations

The lock that a command takes depends on the --scope setting:

--scope value	Lock taken	Effect
worker	One node lease per target node	Multiple worker operations can run in parallel, as long as they target different nodes.
controller+worker	One cluster-wide control_plane_scaling lease	The operation is exclusive. It aborts if any other node operation holds a lease, and it blocks any other node operation while it runs.

For worker scope, you can simultaneously run multiple add and remove operations across different nodes. The following scenarios, however, are not permitted:

• Worker node additions or removals while the cluster state is not stable.
• Concurrent operations that target the same worker node.
• A second remove operation on a worker node already in the process of being removed.

Use the --concurrency flag to control the number of worker nodes that will be processed in parallel within a single command invocation. The default amount is 30.

!!! Control plane (manager) nodes are always joined one at a time, and thus the flag does not affect the k0s bootstrap stage of a control plane addition.

Connectivity

You must have the following capabilities in place to run the mkectl node add and mkectl node remove commands:

• SSH access to each target host, directly or through a bastion host.
• SSH access to one controller node in the cluster, referred to throughout as the helper controller. mkectl selects the first controller that has SSH configured in the MkeConfig object and that is not itself a target of the operation. Bastion access to the controller is also supported.
• Access to the Kubernetes API server for the MKE 4 cluster, using the kubeconfig configured through --kubeconfig.

Note

The helper controller cannot be one of the hosts listed in the hosts file. When you remove control plane nodes, at least one control plane node must remain outside the operation scope to serve as the helper controller.

Control plane requirements

In addition to the constraints described above, a control plane operation verifies the following before it changes anything:

• Every etcd member outside the operation scope is healthy. Adding a control plane node joins it as a voting member, which raises the quorum bar before the new node can contribute, and thus a cluster that merely holds quorum today is not sufficient. Removing a control plane node requires the members that will remain to be healthy; the removal target itself may be unhealthy.
• etcd membership and control plane Node objects agree. Every etcd member (except of the target for the operation), must have corresponding registered control plane Node object.
• Every target host has a usable etcd state. Each host in scope is classified against both sources of truth—the raw etcd member list and the EtcdMember custom resources. For the classifications and their meaning, refer to etcd member verdicts.

Anything else is a hard failure that reports a verdict, a runbook topic, and a link to the etcd membership runbook. mkectl never mutates etcd to repair such a state, as the wrong repair branch is unrecoverable.

Scheduled etcd defragmentation

If scheduled defragmentation is enabled for the etcd maintenance service, you must disable it before you run mkectl node add or mkectl node remove commands with --scope controller+worker. This prevents a scheduled defragmentation from starting while etcd membership is changing and interfering with the operation. mkectl does not currently enforce this requirement.

1. Set defragEnabled to false under spec.etcd.maintenanceService and apply the configuration. Refer to etcd maintenance operations scheduling for more information.

2. On a controller node, verify that no defragmentation is currently running. DefragInProgress must be false:

   curl http://localhost:18088/cluster/status
   

3. Add or remove the control plane nodes.

4. Once the operation completes, set defragEnabled back to true and apply the configuration to restore scheduled defragmentation.

etcd quorum arithmetic

An etcd cluster with an even number of members tolerates no more failures than the odd count below it. Consequently, mkectl node remove aborts if the removal would leave an even number of etcd members. Pass --force to override, in which case the command logs a warning and proceeds.

Backups

A control plane operation that changes etcd membership takes an etcd snapshot of the cluster twice: once before it changes anything, and once after it completes.

• The pre-operation backup is written to the path given by --name, the default setting for which is backup.tar.
• The post-operation backup is written next to it, with a post_ prefix on the file name. For example, --name /mke/dumps/x.tar yields /mke/dumps/post_x.tar.
• Existing files at those paths are overwritten.
• You must either supply --passphrase to encrypt the backups or explicitly opt out with --no-passphrase, unless you pass --skip-snapshot.

The operation skips both backups automatically when it will not change etcd membership. The etcd state verification pre-check determines whether any in-scope host will join or leave etcd. If no host will, the operation skips both the pre-operation and post-operation backups. This occurs, for example, when you re-run the command against control plane nodes that are already joined or already removed. The debug-level log output records the skip. With --dry-run, the backup stage reports a pending backup only when etcd membership will change.

To disable both backups explicitly, pass --skip-snapshot. With this flag set, you do not need to specify --passphrase or --no-passphrase, and the command rejects --passphrase. Use --skip-snapshot only when you have a recent cluster backup to restore from, because the operation does not create a restore point of its own.

Backup flags are ignored for worker scope, as a worker operation does not touch etcd membership and thus takes no snapshot.

Air-gapped environments

The mkectl node add and mkectl node remove commands work in air-gapped environments. When adding a node, mkectl performs an additional verification that the local registry for the cluster contains the required artifacts before proceeding:

• For worker scope, the k0s binary artifact.
• For control plane scope, the k0s binary artifact and the control plane node artifacts, which include the etcd maintenance service artifact that is installed on each new control plane node.

MkeConfig update scope

The mkectl node add and mkectl node remove commands update the MkeConfig object stored in the cluster only for the specific hosts that are listed in the operation. The full hosts list is not reconciled against actual cluster membership.

Warning

Use the mkectl node add and mkectl node remove commands for all changes to cluster membership. Do not use mkectl apply to add or remove nodes. The only exception is Windows worker nodes, which the node commands do not support. To add a Windows worker node, add the host to spec.hosts and run mkectl apply.

The node commands do not update your local mke4.yaml configuration file. After you run them, spec.hosts no longer matches the cluster membership, and mkectl apply behaves as follows:

• Control plane nodes: If the control plane hosts in spec.hosts do not match the control plane nodes of the cluster, the command fails before it changes anything on the cluster.
• Worker nodes: The command joins every worker host in spec.hosts that is not part of the cluster, including a worker node that you removed with mkectl node remove.

Before you run mkectl apply, update spec.hosts to match the current cluster membership. For example, run mkectl config get > mke4.yaml.

Host configuration file

The mkectl node add and mkectl node remove commands both read the list of target hosts from a YAML configuration file. Use the --hosts-path flag to specify the file path, the default setting for which is hosts.yaml.

The file can hold entries of both roles. The --scope flag selects which of them the command acts on, and the remaining entries are ignored.

Example host configuration file:

hosts:
  - role: worker
    installFlags:        # optional
      - "--profile=custom-profile"
    ssh:
      address: 10.0.1.15
      user: root
      keyPath: ~/.ssh/key.pem
      port: 22
      bastion:           # optional
        address: 1.2.3.4
        user: root
        keyPath: ~/.ssh/bastion-key.pem
        port: 22
  - role: worker
    ssh:
      address: 10.0.1.16
      user: root
      keyPath: ~/.ssh/key.pem
      port: 22
  - role: controller+worker
    ssh:
      address: 10.0.1.20
      user: root
      keyPath: ~/.ssh/key.pem
      port: 22

Flags detail

Flags common to both mkectl node add and mkectl node remove

Flag	Description	Default
--scope	Role of the nodes to act on. Selects both: the hosts filter and the execution path. Valid values: worker, controller+worker.	worker
--hosts-path	Path to the hosts configuration file.	hosts.yaml
--timeout-minutes	Total timeout for the operation in minutes. Set to 0 to wait indefinitely.	60
--ssh-key	SSH key file for connecting to the cluster controller. Overrides the key stored in MkeConfig for that controller. Required when the stored key path is absent or not accessible on this machine. Note: The --ssh-key flag does not affect SSH keys for the target nodes as those come from the hosts.yaml configuration file.	—
--bastion-ssh-key	SSH key for the bastion host used to reach the cluster controller. Has no effect if the chosen controller has no bastion configured in MkeConfig. Note: The --bastion-ssh-key flag does not override per-host bastion keys in the hosts.yaml configuration file.	—
--concurrency	Maximum number of nodes to process in parallel. Set to 0 for unlimited. Does not affect the k0s bootstrap stage of a control plane addition, which is always sequential.	30
--dry-run	Simulate the operation without making any changes.	false
--force	Bypass user confirmations. For mkectl node remove --scope=controller+worker, additionally bypasses the even etcd member count check.	false
--collect-facts-timeout	Timeout for the collect-facts stage, which gathers per-host facts over SSH.	5m0s
--lease-ttl	Time To Live (TTL) for the lease objects used to coordinate cluster-wide and node-level operation locks.	1m0s
--lease-renew-interval	The lease renewal frequency. Must be less than that set with the --lease-ttl flag.	20s
--lease-adopt-stale	When set, a matching stale lease is adopted instead of failing with an error.	false

Flags specific to mkectl node add

Flag	Description	Default
--readiness-timeout-minutes	Wait time for each new node to reach Ready state. Set to 0 to wait indefinitely. Must be less than the amount set with the --timeout-minutes flag.	3
--etcd-members-join-timeout-minutes	Total wait time for all new etcd members to join the cluster. Applies to --scope=controller+worker only. Set to 0 to cap the wait by --timeout-minutes instead. Must be less than the amount set with the --timeout-minutes flag.	20
--skip-os-check	Skip the supported-OS-variant check on the target hosts.	false
--skip-storage-check	Skip the storage checks on the target hosts.	false
--skip-cpu-cores-check	Skip the sufficient-CPU-cores check on the target hosts.	false
--skip-total-memory-check	Skip the sufficient-memory check on the target hosts.	false

Flags specific to mkectl node remove

Flag	Description	Default
--drain-timeout-minutes	Wait time for a node drain to complete. Set to 0 to wait indefinitely. Must be less than the amount set with the --timeout-minutes flag.	3
--drain-grace-period-minutes	Grace period given to each Pod to terminate gracefully. If negative, the default grace period for the Pod is used. Must be less than the amount set with the --drain-timeout-minutes flag.	1.5
--etcd-member-leave-timeout-minutes	Wait time per etcd member to leave the cluster. Applies to --scope=controller+worker only. Set to 0 to cap the wait by --timeout-minutes instead. Must be less than the amount set with the --timeout-minutes flag.	3

Backup flags (control plane scope only)

The following flags configure the pre- and post-operation backups that a control plane operation takes. They have no effect when --scope=worker. For the conditions under which the backups are taken or skipped, refer to Backups.

Flag	Description	Default
--skip-snapshot	Skip both the pre- and the post-operation backup. Cannot be combined with --passphrase. When set, neither --passphrase nor --no-passphrase is required.	false
--name	Path of the pre-operation backup file. The post-operation backup is written to the same directory with a post_ prefix on the file name.	backup.tar
--passphrase	Encrypt the backup files with a passphrase of at least 12 characters. Use - to read the passphrase from stdin instead of the command line. Rejected when --skip-snapshot is set.	—
--no-passphrase	Opt out of encrypting the backup files. Not recommended. Mutually exclusive with --passphrase; one of the two is required unless --skip-snapshot is set.	false
--include-logs	Write a sidecar <name>.log next to each backup, capturing the backup run's operation log. The log is not encrypted.	false
--progress-bars	Display progress bars during snapshot creation.	true
--wait	If another etcd maintenance operation holds the node's operation lock—defragmentation, cleanup, quota recovery, or a scheduled backup—wait for it to be released instead of failing fast.	false