Add nodes#

The mkectl node add command bootstraps k0s on each target node, joins it to the cluster, and updates the MkeConfig stored in the cluster to include the new node.

The --scope flag selects the role to add. With --scope=worker, the default, the command adds worker nodes. With --scope=controller+worker, it adds control plane nodes, which additionally verify etcd membership, take pre- and post-operation cluster backups whenever etcd membership changes, and scale the MKE addons onto the new control plane nodes. For more information, refer to Node roles and the --scope flag.

Note

The mkectl node add command is idempotent. When you run it against hosts that have already joined the cluster, mkectl skips every stage that has no host to act on:

The k0s bootstrap does not run.
The control plane configuration sync copies no files.
For control plane nodes, mkectl takes no backup, because etcd membership does not change.

Only the node readiness verification and the MkeConfig patch run, and neither changes the cluster state.

Command action sequences#

Both roles start with the same set of checks and then diverge. The command action stages whose sequences are described herein run for each host in the hosts.yaml configuration file that matches the --scope setting.

Checks common to both roles#

The mkectl node add command runs the following checks first, in the order given, for either role. None of these checks change the cluster.

Reachability check.

mkectl verifies SSH connectivity to the target hosts and to the helper controller, which is the existing control plane node through which the cluster is inspected.
Host state detection.

mkectl inspects each host to determine whether k0s is installed and running, whether the host belongs to this cluster or to another one, and whether its node is registered and Ready. This determines the path taken for that host. For the full matrix, refer to Node scenarios.

A host that belongs to a different cluster will cause the entire operation to fail.
Fact collection.

Per-host facts, such as the cgroup version, are gathered through SSH. The timeout is set with the --collect-facts-timeout flag.
Pre-flight checks.

Host-level checks—supported OS variant, storage, CPU cores, total memory, and cgroup v2—run against the hosts that are about to receive a fresh k0s install. The first four can be disabled individually; refer to Flags specific to mkectl node add.

This stage also runs the components version check: the command aborts if the MKE 4 cluster was deployed with a different mkectl binary or if the cluster runs a different k0s version than the version that mkectl supports.
Registry verification.

Registry credentials are read back from the credentialsSecretName of each registry in MkeConfig, and mkectl verifies that the registry holds the artifacts that are required for the operation. For more information, refer to Air-gapped environments.
Cluster state check.

The mkectl node add command aborts if the cluster is not stable. All cluster nodes that are not currently in a transitioning state must be in Ready state.
Etcd cluster state check.

mkectl aborts if etcd is not operational. For the exact conditions, refer to Cluster state.
Operation lock.

For worker scope, each target node is marked as adding to prevent concurrent operations from targeting the same node. For control plane scope, a single cluster-wide lock is taken instead, and the command aborts if any other node operation/cluster-wide operation is in progress. For more information, refer to Concurrent operations.

Worker node addition#

Once the common checks pass, the worker node addition sequence commences as follows:

Registry CA sync.

The registry CA certificate is copied to each new worker.
k0s binary staging.

The k0s binary is downloaded from the cluster registry onto each new worker.
SELinux labels.

Where SELinux is enabled, the required labels are applied to the new hosts.
Worker bootstrap.

k0s is installed and configured on the node, which then joins the MKE 4 cluster. Existing controllers are not touched.

Note

The worker bootstrap stage is skipped if no host in scope requires a fresh k0s install.
Service restart.

On a host that already runs healthy k0s but whose node never registered in the cluster, the k0s worker service is restarted so that registration is retried.
Cloud provider reinstall.

Where the cluster is configured with an external cloud provider, k0s is reinstalled on the new worker with the corresponding flags.
Un-cordon.

If a re-joining node reused a stale, cordoned Node object, it is uncordoned. This is a no-op on an already schedulable node.
Verification.

mkectl waits for the node to appear in the cluster and reach Ready state within the amount set by the --readiness-timeout-minutes flag.
Node label reconciliation.

User-declared node labels from installFlags are reconciled onto the Node object, and the mkectl ownership annotation is recorded so that a later mkectl apply can reconcile those labels.
MkeConfig config patch.

The MkeConfig stored in the cluster is updated to include the new node.

Control plane node addition#

Once the common checks pass, the control plane node addition sequence commences as follows:

Stale EtcdMember CR cleanup.

An EtcdMember custom resource left behind by a previous removal of a host now in scope is deleted, so that the etcd state verification considers a clean host as being clean.
Etcd state verification (pre-check).

Each host in scope is classified against both sources of truth: the raw etcd member list read on the helper controller and the EtcdMember custom resources. The result is checked against the requirements in Control plane requirements. The observed member set is recorded as the baseline that the post-check compares against. The pre-check also records whether the operation is going to change etcd membership, which is the case when at least one host in scope is not yet an etcd member. This determines whether the pre- and post-operation backups are taken.

This stage never mutates etcd. Anything it rejects is a hard failure that reports a verdict and a runbook topic. Refer to etcd member verdicts and to the etcd membership runbook for more information.
Pre-operation backup.

An etcd snapshot of the cluster is written to the path provided by the --name option. The stage is skipped if the pre-check determined that no host in scope is going to join etcd, or if --skip-snapshot is set. For more information, refer to Backups.
Control plane configuration sync.

The configuration and certificate material that is required by a control plane node is copied to the new hosts: the API server user and its certificates, the admission control and event rate limit configuration, the encryption configuration, the audit configuration, the client CA, the OIDC authentication configuration, and the registry CA. Material that only the existing cluster holds is read from the helper controller.
k0s binary staging.

The k0s binary is downloaded from the cluster registry onto each new control plane host.
SELinux labels.

Where SELinux is enabled, the required labels are applied to the new hosts.
Control plane bootstrap.

k0s is installed and configured on the node, which then joins the MKE 4 cluster as a control plane node and as a voting etcd member. Control plane nodes are joined one at a time, regardless of the --concurrency setting, and existing controllers and workers are left unchanged. If no host in scope requires a fresh k0s install, this stage is skipped.
API server wait.

mkectl waits up to 10 minutes for the Kubernetes API server to become reachable again, as a joining control plane node briefly disrupts it.
Un-cordon.

If a re-joining node reused a stale, cordoned Node object, it is uncordoned. This is a no-op on an already schedulable node.
Etcd member join wait.

mkectl waits for every new host to display in the etcd member list, within the amount set by the --etcd-members-join-timeout-minutes flag.
Service restart.

On a host that already runs healthy k0s but whose node never registered in the cluster, the k0s controller service is restarted so that registration is retried.
Verification.

mkectl waits for the node to appear in the cluster and reach Ready state within the amount set by the --readiness-timeout-minutes flag.
Etcd state verification (post-check).

Every member recorded in the pre-check baseline must still be present and healthy, and every new host must now be a healthy etcd member. A host that is registered but whose etcd join never completed is reported at this point.
MkeRelease readiness wait.

Joining control plane nodes disrupts the webhook endpoints that the MkeRelease reconciliation depends on, which makes it briefly not Ready. Because the MkeConfig write below goes through a validating webhook that resolves spec.version to a Ready MkeRelease, mkectl waits out that window first.
MkeConfig patch.

The MkeConfig stored in the cluster is updated to include the new node. This also signals the MKE operator to scale the addon replica counts to the new control plane count.
Node label reconciliation.

User-declared node labels from installFlags are reconciled onto the Node object, and the mkectl ownership annotation is recorded.
Etcd maintenance service installation.

The etcd maintenance service is installed on the new control plane nodes.
Addon scale-out wait.

mkectl waits for the MKE addons to finish scaling onto the new control plane nodes.
Workload redistribution.

mkectl restarts the MKE-managed Deployments and StatefulSets in the mke and k0rdent namespaces to spread the MKE system components across all control plane nodes. As with the backup stages, mkectl skips this stage if the pre-check finds that no host in scope will join etcd. For more information, refer to Workload redistribution.
Post-operation backup.

A second etcd snapshot, with the post_ file name prefix, is written to the same directory as the first. As with the pre-operation backup, this stage is skipped if no etcd membership change was made or if --skip-snapshot is set.

Failure handling#

Upon failure, the operation lock is cleared. In this event, as the operation is idempotent, Mirantis recommends that you retry the command. If the operation continues to fail, manually clean up the node and rerun the mkectl node add command again.

For control plane scope, a failure reported with an etcd verdict and a runbook topic is not a candidate for a blind retry: mkectl deliberately does not repair etcd membership on its own, because the wrong repair branch is unrecoverable. Resolve the reported state first, using etcd member verdicts and the etcd membership runbook, and only then attempt a retry.

Examples: `mkectl node add`#

Command example	Description
`mkectl node add --hosts-path hosts.yaml`	Add the worker nodes defined in `hosts.yaml`. Worker is the default scope.
`mkectl node add --hosts-path hosts.yaml \` `--scope=controller+worker --no-passphrase`	Add the control plane nodes defined in `hosts.yaml`, with unencrypted pre- and post-operation backups.
`mkectl node add --hosts-path hosts.yaml \` `--scope=controller+worker --passphrase - \` `--name /mke/dumps/cp-add.tar`	Add control plane nodes, reading the backup passphrase from stdin and writing the backups to `/mke/dumps/cp-add.tar` and `/mke/dumps/post_cp-add.tar`.
`mkectl node add --hosts-path hosts.yaml \` `--scope=controller+worker --skip-snapshot`	Add control plane nodes without taking the pre- and post-operation backups.
`mkectl node add --hosts-path hosts.yaml \` `--timeout-minutes 20`	Add nodes with a custom total timeout of 20 minutes.
`mkectl node add --hosts-path hosts.yaml \` `--readiness-timeout-minutes 5`	Add nodes with a custom node readiness timeout of 5 minutes.
`mkectl node add --hosts-path hosts.yaml \` `--scope=controller+worker --no-passphrase \` `--etcd-members-join-timeout-minutes 30`	Add control plane nodes, allowing 30 minutes in total for the new etcd members to join.
`mkectl node add --hosts-path hosts.yaml \` `--ssh-key ~/.ssh/prod-key.pem`	Add nodes using an SSH key to connect to the helper controller.
`mkectl node add --hosts-path hosts.yaml \` `--ssh-key ~/.ssh/node-key.pem \` `--bastion-ssh-key ~/.ssh/bastion-key.pem`	Add nodes when the helper controller is behind a bastion host.