Skip to content

Remove worker nodes#

The mkectl node remove command cordons off and drains the node targeted for removal, removes that node from the MKE 4 cluster, runs k0s reset on the host, cleans up various folders and files to remove any traces of k0s, and updates the MkeConfig stored in the cluster to remove the node.

Note

The mkectl node remove command is idempotent, and thus running it against hosts that are not part of the cluster is a no-op.

Conmand action sequence#

The worker node removal sequence for each host in the hosts.yaml configuration file is as follows:

  1. Reachability check.

    mkectl verifies SSH connectivity to the target host and to the cluster controller.

  2. Cluster state check.

    The mkectl node remove command aborts if the MKE 4 cluster is not stable. All cluster nodes that are not currently in a transitioning state must be in Ready state.

  3. Components version check.

    The mkectl node remove command aborts if the cluster was deployed with a different mkectl binary or if the cluster runs a different k0s version than the one mkectl supports.

  4. Node state check.

    mkectl determines whether the node exists in the cluster and its health status, to decide which removal path to take. Refer to Node scenarios for the full matrix.

  5. Node state lock.

    The node is marked as removing.

  6. Cordon.

    The node is marked unschedulable. No new workloads are scheduled on it.

  7. Drain.

    Workloads are evicted from the node. For that, the kubectl drain command is executed against it, run on a controller node:

    kubectl drain --force \
  --grace-period=<seconds> \
  --timeout=<duration> \
  --delete-emptydir-data \
  --ignore-daemonsets \
  <node-name>

    Warning

    If the drain operation fails, the error is reported and node removal continues. The node is still deleted from the cluster.

  8. Delete node.

    The node object is removed from the Kubernetes cluster.

  9. Cleanup.

    k0s reset is run on the host to remove MKE 4 artifacts. In addition, the command removes any lingering traces of k0s from various files and folders.

    Warning

    If you receive an error indicating that the cleanup operation has failed, the node has already been removed from the cluster, however the host may require manual cleanup. To confirm, re-run mkectl node remove. If the error persists, manual host cleanup is required.

  10. Cluster config patch.

    The MkeConfig stored in the cluster is updated to remove the node.

Examples: mkectl node remove#

Command example Description
mkectl node remove --hosts-path hosts.yaml Remove nodes defined in the hosts.yaml configuration file.
mkectl node remove --hosts-path hosts.yaml --timeout-minutes 20 Remove nodes with a custom total timeout of 20 minutes.
mkectl node remove --hosts-path hosts.yaml\
    --drain-timeout-minutes 5 \
    --drain-grace-period-minutes 2.5		 Remove nodes with a custom drain timeout of 5 minutes and a grace period of 2.5 minutes.
mkectl node remove --hosts-path hosts.yaml --ssh-key ~/.ssh/prod-key.pem Remove nodes using an SSH key to connect to the cluster controller.
mkectl node remove --hosts-path hosts.yaml\
     --ssh-key ~/.ssh/node-key.pem \
     --bastion-ssh-key ~/.ssh/bastion-key.pem		 Remove nodes when the controller is behind a bastion host.