Scale worker nodesNew

You can add and remove worker nodes using the mkectl node add and mkectl node remove commands, respectively, from a running MKE 4 cluster without having to perform a full cluster reconciliation. Each command runs a targeted sequence of steps, bootstrap or cordon/drain/reset, and updates the MkeConfig stored in the cluster to reflect the new node membership.

Limitations

• Only worker nodes are supported. You cannot use the instructions herein to add or remove control plane nodes.
• You must provision target nodes in advance. Cloud and virtual machines resources are not automatically created.
• Automatic recovery does not occur if an operation is interrupted before completion, and thus you may need to manually clean up the affected node.
• Windows worker nodes are not supported.
• The commands update MkeConfig only for the hosts explicitly listed in the operation. The hosts list is not otherwise reconciled against actual cluster membership. For more information, refer to MkeConfig update scope.

Prerequisites and constraints

Version alignment

The mkectl binary version must match the version that was used to deploy your MKE 4 cluster. The k0s version that mkectl installs on new worker nodes must also match the k0s version running on existing cluster nodes.

Cluster state

MKE 4 tracks cluster-wide state-to-gate node scaling. Before executing, the mkectl node add and mkectl node remove commands both verify that the cluster is in a stable state and aborts if it is not.

Operations that place the cluster in a non-stable state—such as upgrade, apply, and reset—will block node scaling until they complete. Cluster state is managed using lease objects that are stored in the cluster, in the mke namespace by default.

Node state

Each node carries an operation signal that indicates whether it is being added, being removed, or remaining idle. Node state is managed through lease objects, which prevent concurrent conflicting operations on the same node; for example, attempting to simultaneously add and remove the same node.

Concurrent operations

You can simulataniously run multiple add and remove operations across different nodes. The following scenarios, however, are not permitted:

• Node additions or removals while the cluster state is not stable.
• Concurrent operations that target the same node.
• A second remove operation on a node already in the process of being removed.

Use the --concurrency flag to control the number of nodes that will be processed in parallel within a single command invocation. The default amount is 30.

Connectivity

You must have the following capabilities in place to run the mkectl node add and mkectl node remove commands:

• SSH access to each target host, directly or through a bastion host.
• SSH access to one controller node in the cluster. mkectl selects the first controller that has SSH configured in the MkeConfig object. Bastion access to the controller is also supported.
• Access to the Kubernetes API server for the MKE 4 cluster, using the kubeconfig configured through --kubeconfig.

Air-gapped environments

The mkectl node add and mkectl node remove commands work in air-gapped environments. When adding a node, mkectl performs an additional verifies that the local registry for the cluster contains the required k0s binary artifact before proceeding.

MkeConfig update scope

The mkectl node add and mkectl node remove commands update the MkeConfig object stored in the cluster only for the specific hosts that are listed in the operation. The full hosts list is not reconciled against actual cluster membership.

Warning

If mkectl apply is later run with a spec.hosts section that does not reflect the current cluster membership — for example, after nodes have been added or removed with mkectl node add or mkectl node remove -- the command reconciles the cluster to match spec.hosts, and that can cause unintended changes if the hosts list is stale.

Host configuration file

The mkectl node add and mkectl node remove commands both read the list of target hosts from a YAML configuration file. Use the --hosts-path flag to specify the file path, the default setting for which is hosts.yaml.

Example host configuration file:

hosts:
  - role: worker
    ssh:
      address: 10.0.1.15
      user: root
      keyPath: ~/.ssh/key.pem
      port: 22
      installFlags:        # optional
        - "--profile=custom-profile"
      bastion:             # optional
        address: 1.2.3.4
        user: root
        keyPath: ~/.ssh/bastion-key.pem
        port: 22
  - role: worker
    ssh:
      address: 10.0.1.16
      user: root
      keyPath: ~/.ssh/key.pem
      port: 22

Flags detail

Flags common to both mkectle node add and mkectl node remove

Flag	Description	Default
--hosts-path	Path to the hosts configuration file.	hosts.yaml
--timeout-minutes	Total timeout for the operation in minutes. Set to 0 to wait indefinitely.	60
--ssh-key	SSH key file for connecting to the cluster controller. Overrides the key stored in MkeConfig for that controller. Required when the stored key path is absent or not accessible on this machine. Note: The --ssh-key flag does not affect SSH keys for the target worker nodes as those come from the hosts.yaml configuration file.	—
--bastion-ssh-key	SSH key for the bastion host used to reach the cluster controller. Has no effect if the chosen controller has no bastion configured in MkeConfig. Note: The --bastion-ssh-key flag does not override per-worker bastion keys in the hosts.yaml configuration file.	—
--concurrency	Maximum number of nodes to process in parallel. Set to 0 for unlimited.	30
--dry-run	Simulate the operation without making any changes.	false
--lease-ttl	Time To Live (TTL) for the lease objects used to coordinate cluster-wide and node-level operation locks.	1m0s
--lease-renew-interval	The lease renewal frequency. Must be less than that set with the --lease-ttl flag.	20s
--lease-adopt-stale	When set, a matching stale lease is adopted instead of failing with an error.	false

Flags specific to mkectl node add

Flag	Description	Default
--readiness-timeout-minutes	Wait time for each new node to reach Ready state. Set to 0 to wait indefinitely. Must be less than the amount set with the --timeout-minutes flag.	3

Flags specific to mkectl node remove

Flag	Description	Default
--drain-timeout-minutes	Wait time for a node drain to complete. Set to 0 to wait indefinitely. Must be less than the amount set with the --timeout-minutes flag.	3
--drain-grace-period-minutes	Grace period given to each Pod to terminate gracefully. If negative, the default grace period for the Pod is used.	1.5