mkectl node remove
mkectl node remove#
Remove one or more manager/worker nodes from an existing MKE cluster.
Synopsis#
The operation cordons, drains, deletes node(s), leaves etcd (for managers only), stops, resets k0s on each target host, and updates the MKEConfig in cluster. The command is idempotent - if run against the list of hosts that already removed from the cluster - no-op.
Prerequisites: 1. The cluster must be in a stable state (no upgrade, apply, or reset in progress).
-
Hosts that are to be removed from the cluster must not be a member of any k8s cluster except of the current. No active operations (adding/removing to/from cluster) are executed on the hosts.
-
Etcd cluster should be operational
Note on draining process:
The command uses kubectl drain
The kubectl command is run for every manager/worker node, with these flags:
kubectl drain --force --grace-period=
mkectl node remove [flags]
Examples#
# Remove worker nodes defined in hosts.yaml
mkectl node remove --hosts-path hosts.yaml
# Remove manager nodes defined in hosts.yaml, backup (pre and post) is created without passphrase
mkectl node remove --hosts-path hosts.yaml --scope=controller+worker --no-passphrase
# Remove worker nodes with a custom total timeout
mkectl node remove --hosts-path hosts.yaml --timeout-minutes 20
# Remove manager nodes with a custom node drain timeout and grace period,
# backup (pre and post) is created without passphrase
mkectl node remove --hosts-path hosts.yaml --scope=controller+worker \
--no-passphrase --drain-timeout-minutes 5 --drain-grace-period-minutes 2.5
# Remove worker nodes using SSH key from file to connect to controller
mkectl node remove --hosts-path hosts.yaml --ssh-key ~/.ssh/prod-key.pem
# Remove worker nodes, when accessing controller require using bastion
# jump host (with SSH key from file for accessing bastion)
mkectl node remove --hosts-path hosts.yaml --ssh-key ~/.ssh/node-key.pem \
--bastion-ssh-key ~/.ssh/bastion-key.pem
Host config file format:
hosts:
- role: worker # worker/controller+worker
installFlags: # optional
- "--profile=custom-profile"
ssh:
address: 10.0.1.15
user: root
keyPath: ~/.ssh/key.pem
port: 22
bastion: # optional
address: 1.2.3.4
user: root
keyPath: ~/.ssh/bastion-key.pem
port: 22
Options#
--drain-grace-period-minutes float Period of time given to each node's pod to terminate gracefully. Timeout should be less than '--drain-timeout-minutes'. If negative, the default value specified in the pod will be used. (default 1.5)
--drain-timeout-minutes int Specify how long to wait for drain node. Timeout should be less than '--timeout-minutes' for operation. If 0, then wait indefinitely. (default 3)
--etcd-member-leave-timeout-minutes int Specify how long to wait per etcd member to leave the cluster. Used only for removing control planes. This timeout multiplied by members count should be less than '--timeout-minutes' for operation (not enforced by mkectl). If 0, then wait is capped by '--timeout-minutes'. (default 3)
-h, --help help for remove
Options inherited from parent commands#
--bastion-ssh-key string Specify the ssh key file for bastion host to be used to connect to existing cluster's controller host
(when MKEConfig specifies one for that controller). Controller is picked from MKEConfig as first entry
with configured SSH. Same precedence and scope as --ssh-key. Has no effect when the chosen controller
has no bastion configured in MKEConfig.
Does NOT override the per-worker/cp bastion key in hosts.yaml.
--collect-facts-timeout duration Timeout for the collect-facts phase (gathering per-host facts such as cgroup version over SSH) (default 5m0s)
--concurrency int Maximum number of nodes to configure in parallel, set to 0 for unlimited (default 30)
--dry-run Simulate the node operations without making any changes
-f, --file string path to the config file. Not needed for restore where config is loaded from the backup (default "mke4.yaml")
--force Bypass user confirmations. This will apply changes without any confirmations
--hosts-path string path to the hosts config file (default "hosts.yaml")
--include-logs Write a sidecar <name>.log next to the backup capturing the backup run's operation log (the log is not encrypted)
--kubeconfig string path to the kubeconfig file of the existing or restored cluster. DO NOT USE when creating a new cluster (default "/home/runner/.mke/mke.kubeconf")
--lease-adopt-stale if true, matching stale lease would be adopted instead of failing with error
--lease-renew-interval duration lease renew interval for cluster-wide/node operations, should be < leaseTTL (default 20s)
--lease-ttl duration lease TTL for cluster-wide/node operations (default 1m0s)
-l, --logLevel string specify a log level (info, warn, debug, trace, error) (default "info")
--name string Name of the snapshot to create (default "backup.tar")
--no-color when provided, disables colored output
--no-passphrase Opt out to encrypt the tar file with a passphrase (not recommended)
--no-pretty-print when provided, disables pretty print output formatting
--passphrase string Encrypt the tar file with a passphrase (at least 12 characters). Use "-" to read it from stdin instead of the command line
--progress-bars Enable progress bars during snapshot creation (default true)
--scope string The scope of the command. Valid values: 'worker,controller+worker'. This defines the role based filter for entries in hosts.yaml. The filtered list will be used as target hosts for the command. (default "worker")
--skip-snapshot Skip taking etcd-snapshot before and after adding/removal managers. Not used for workers scope.
--ssh-key string Specify the ssh key file to be used to connect to existing cluster's controller host. Controller is picked from
MKEConfig as first entry with configured SSH. If specified, overrides the keyPath stored in MKEConfig
for that controller; the file must exist locally. Required when MKEConfig's controller keyPath is empty
or not present on this machine.
Does NOT affect SSH keys for the new workers/cp nodes being added — those come from hosts.yaml only.
--temp-dir string Directory on the controller where k0s stages the etcd snapshot while the backup is taken; it needs free space at least the size of the etcd database. Overrides spec.backup.tempDir; when both are empty k0s uses /tmp
--timeout-minutes int Use specified timeout (in minutes) for node operations. If 0, then wait indefinitely. (default 60)
--version version[=true] --version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version
--wait If the node's etcd-maintenance operation lock is held by another operation (defrag, cleanup, quota recovery, or a scheduled backup), wait for it to be released instead of failing fast