Remove nodes#

The mkectl node remove command cordons off and drains the node targeted for removal, removes that node from the MKE 4 cluster, runs k0s reset on the host, cleans up various folders and files to remove any traces of k0s, and updates the MkeConfig stored in the cluster to remove the node.

The --scope flag selects the role to remove. With --scope=worker, the default, the command removes worker nodes. With --scope=controller+worker, it removes control plane nodes, which additionally makes each target leave the etcd cluster and takes pre- and post-operation cluster backups whenever etcd membership changes. For more information, refer to Node roles and the --scope flag.

Note

The mkectl node remove command is idempotent. When you re-run it against hosts that are no longer part of the cluster, the command skips every stage that has no host to act on. For control plane scope, it also skips the etcd backup, because etcd membership does not change. Stages that do not target specific hosts, such as the MkeConfig patch, still run but leave the cluster in the same state.

Warning

At least one control plane node must remain outside the operation scope, to serve as the helper controller through which the removal is driven.

Command action sequences#

Both roles start with the same set of checks and then diverge. The command action stages whose sequences are described herein run for each host in the hosts.yaml configuration file that matches the --scope setting.

Checks common to both roles#

The mkectl node remove command runs the following checks first, in the order given, for either role. None of them changes the cluster.

Reachability check.

mkectl verifies SSH connectivity to the target hosts and to the helper controller, which is the existing control plane node through which the cluster is inspected and modified.
k0s and Node state check.

mkectl determines whether each node exists in the cluster and what its health status is, to ascertain which removal path to take. For the full matrix, refer to Node scenarios.

A host that belongs to a different cluster will cause the entire operation to fail. A host that is neither part of the cluster nor has k0s installed is reported and excluded.
Fact collection.

Per-host facts are gathered over SSH. The timeout is set using the --collect-facts-timeout flag.
Components version check.

The mkectl node remove command aborts if the cluster was deployed with a different mkectl binary or if the cluster runs a different k0s version than the one mkectl supports.

The host capacity checks that mkectl node add runs -- OS variant, storage, CPU cores, memory, and cgroup version -- are not run on a removal.
Cluster state check.

The mkectl node remove command aborts if the MKE 4 cluster is not stable. All cluster nodes that are not currently in a transitioning state must be in Ready state.
Etcd cluster state check.

mkectl aborts if etcd is not operational. For information on the exact conditions, refer to Cluster state.
Operation lock.

For worker scope, each target node is marked as removing. For control plane scope, a single cluster-wide lock is taken instead, and the command aborts if any other node operation is in progress. For more information, refer to Concurrent operations.

Worker node removal#

Once the common checks above pass, the worker node removal sequence is as follows:

Cordon.

The node is marked unschedulable, and thus no new workloads are scheduled on it.
Drain.

Workloads are evicted from the node. For that, the kubectl drain command is executed against it, run on a controller node:
```
kubectl drain --force \
  --grace-period=<seconds> \
  --timeout=<duration> \
  --delete-emptydir-data \
  --ignore-daemonsets \
  <node-name>
```
Warning

If the drain operation fails, the error is reported and node removal continues. The node is still deleted from the cluster.
Delete node.

The node object is removed from the Kubernetes cluster.
Cleanup.

k0s reset is run on the host to remove MKE 4 artifacts. Hosts whose k0s installation is broken, or whose node never registered in the cluster, are reset in a second pass that skips the drain and delete steps.

Warning

If you receive an error indicating that the cleanup operation has failed, the node has already been removed from the cluster, however the host may require manual cleanup. To confirm, re-run mkectl node remove. If the error persists a manual host cleanup is required.
Leftover cleanup.

Any lingering files and folders that k0s reset left behind are removed.
Cluster config patch.

The MkeConfig stored in the cluster is updated to remove the node.

Control plane node removal#

Once the common checks above pass, the control plane node removal sequence is as follows:

Stale EtcdMember CR cleanup.

An EtcdMember custom resource that outlived the member it describes is deleted for each host in scope, so that the etcd state verification classifies the host on current facts.
Etcd state verification (pre-check).

Each host in scope is classified against both sources of truth: the raw etcd member list read on the helper controller, and the EtcdMember custom resources.

Unlike an addition, only the members that will remain must be healthy; the removal target itself may be unhealthy. The stage also checks the resulting member count and aborts if the removal would leave an even number of etcd members, unless --force is passed. For more information, refer to etcd quorum arithmetic.

This stage never mutates etcd. A target whose etcd state needs manual intervention first is a hard failure that reports a verdict and a runbook topic; refer to etcd member verdicts and to the etcd membership runbook.

The pre-check also records whether the operation is going to change etcd membership, which is the case when at least one target is still a healthy etcd member or a member whose leave has already been requested. This determines whether the pre- and post-operation backups are taken.
Pre-operation backup.

An etcd snapshot of the cluster is written to the path given by --name. This stage is skipped if the pre-check determined that no target will leave etcd, or if --skip-snapshot is set. For more information, refer to Backups.
Cordon and drain.

Every target node is cordoned and drained, in parallel (capped with --concurrency), on a best-effort basis, using the same kubectl drain invocation shown for worker node removal. A failed drain is reported and the removal continues.
Etcd leave and reset.

For each target, in this order: the member is removed from etcd by member ID, issued against the helper controller etcd while the target is still a live voter; k0s is stopped on the target; the EtcdMember custom resource for the target is marked to leave and mkectl waits for the leave to complete, within the amount set by the --etcd-member-leave-timeout-minutes flag; then k0s reset runs on the host and the etcd maintenance service is uninstalled.

k0s reset is invoked with the data directory and the kubelet root directory that the cluster uses, so a custom data directory and the kubelet root directory (/var/lib/kubelet by default) are wiped as well, and a later re-install on the same host is not blocked by leftovers.

Targets are reset in two passes:
- Best-effort pass: Covers hosts that have a broken or unhealthy k0s installation, and hosts that have no k0s installed but are still registered in the cluster. If k0s reset cannot start on a host, for example because the SSH connection is lost, the error is logged and the removal proceeds.
- Strict pass: Covers hosts that have a healthy k0s installation and belong to the cluster. If k0s reset cannot start on a host, the operation fails.
In both passes, he following conditions are reported as warnings rather than errors, because the node is already torn down and its etcd leave is already committed:
- The k0s service fails to stop.
- k0s reset exits with a non-zero code.
The leftover file cleanup stage removes anything that k0s reset leaves behind.

Hosts that are neither part of the cluster nor have k0s installed are not reset, as they were excluded by the common checks.
API server wait.

mkectl waits up to 10 minutes for the Kubernetes API server to become reachable again, as removing a control plane node disrupts it.
Etcd member leave wait.

As a safety net, mkectl confirms that every target has disappeared from the etcd member list.
Delete node.

The node objects of the targets are removed from the Kubernetes cluster.
Etcd state verification (post-check).

Confirm that every member recorded in the pre-check baseline is still present and healthy, and that no target node remains an etcd member. If a target is still a voter after removal, this check reports it.
Addon and MkeRelease readiness wait.

Removing control plane nodes evicts the MKE operator pods. Until they reschedule onto a surviving control plane node, the operator webhook has no ready endpoints and every MkeConfig update is rejected. mkectl therefore waits for the addons to become ready, and then for the MkeRelease to become Ready again, before writing to MkeConfig.
MkeConfig patch.

The MkeConfig stored in the cluster is updated to remove the node.
Leftover EtcdMember CR cleanup.

Any EtcdMember custom resource left for a removed target is deleted.
Leftover file cleanup.

Any lingering files and folders that k0s reset left behind are removed.
Post-operation backup.

A second etcd snapshot is written next to the first one, with a post_ prefix on the file name. The stage is skipped under the same conditions as the pre-operation backup: no etcd membership change was expected, or --skip-snapshot is set.

Failure handling#

Upon failure, the operation lock is cleared, and the command can be retried, as the operation is idempotent.

For control plane scope, note that the removal commits per host: a target whose etcd member was already removed and whose k0s was already reset stays removed even if a later stage fails. A retry picks up from the current state. A failure reported with an etcd verdict and a runbook topic is not a candidate for a blind retry — resolve the reported state first, using etcd member verdicts and the etcd membership runbook.

Examples: `mkectl node remove`#

Command example	Description
`mkectl node remove --hosts-path hosts.yaml`	Remove the worker nodes defined in the `hosts.yaml` configuration file. Worker is the default scope.
`mkectl node remove --hosts-path hosts.yaml \` `--scope=controller+worker --no-passphrase`	Remove the control plane nodes defined in `hosts.yaml`, with unencrypted pre- and post-operation backups.
`mkectl node remove --hosts-path hosts.yaml \` `--scope=controller+worker --no-passphrase --force`	Remove control plane nodes even though the cluster would be left with an even number of etcd members.
`mkectl node remove --hosts-path hosts.yaml \` `--scope=controller+worker --skip-snapshot`	Remove control plane nodes without taking the pre- and post-operation backups.
`mkectl node remove --hosts-path hosts.yaml --timeout-minutes 20`	Remove nodes with a custom total timeout of 20 minutes.
`mkectl node remove --hosts-path hosts.yaml \` `--drain-timeout-minutes 5 \` `--drain-grace-period-minutes 2.5`	Remove nodes with a custom drain timeout of 5 minutes and a grace period of 2.5 minutes.
`mkectl node remove --hosts-path hosts.yaml \` `--scope=controller+worker --no-passphrase \` `--etcd-member-leave-timeout-minutes 5`	Remove control plane nodes, allowing 5 minutes per etcd member to leave.
`mkectl node remove --hosts-path hosts.yaml --ssh-key ~/.ssh/prod-key.pem`	Remove nodes using an SSH key to connect to the helper controller.
`mkectl node remove --hosts-path hosts.yaml \` `--ssh-key ~/.ssh/node-key.pem \` `--bastion-ssh-key ~/.ssh/bastion-key.pem`	Remove nodes when the helper controller is behind a bastion host.