mkectl node remove

mkectl node remove#

Remove one or more worker nodes from an existing MKE cluster.

Synopsis#

The operation cordons, drains, deletes node(s), stops, resets k0s on each target host, and updates the MKEConfig in cluster. The command is idempotent - if run against the list of hosts that already removed from the cluster - no-op.

Prerequisites: 1. The cluster must be in a stable state (no upgrade, apply, or reset in progress).

Hosts that are to be removed from the cluster must not be a member of any k8s cluster except of the current. No active operations (adding/removing to/from cluster) are executed on the hosts.

Note on draining process: The command uses kubectl drain call under the hood, that is actually: - mark the node as unschedulable (cordon) - evict pods gracefully - continue if it finds (always using --force flag): * unmanaged pods * mirror/static pods * DaemonSet pods (ignored by default with mkectl node remove)

The kubectl command is run for every worker node, with these flags: kubectl drain --force --grace-period= --timeout= \ --delete-emptydir-data --ignore-daemonsets If kubectl drain fails, the node is still removed from the cluster.

mkectl node remove [flags]

Examples#

# Remove nodes defined in hosts.yaml
mkectl node remove --hosts-path hosts.yaml

# Remove nodes with a custom total timeout
mkectl node remove --hosts-path hosts.yaml --timeout-minutes 20

# Remove nodes with a custom node drain timeout and grace period
mkectl node remove --hosts-path hosts.yaml --drain-timeout-minutes 5 --drain-grace-period-minutes 2.5

# Remove nodes using SSH key from file to connect to controller
mkectl node remove --hosts-path hosts.yaml --ssh-key ~/.ssh/prod-key.pem

# Remove nodes, when accessing controller require using bastion jump host (with SSH key from file for accessing bastion) 
mkectl node remove --hosts-path hosts.yaml --ssh-key ~/.ssh/node-key.pem --bastion-ssh-key ~/.ssh/bastion-key.pem

Host config file format:

hosts:
  - role: worker
    installFlags: # optional
      - "--profile=custom-profile"
    ssh:
      address: 10.0.1.15
      user: root
      keyPath: ~/.ssh/key.pem
      port: 22
      bastion: # optional
        address: 1.2.3.4
        user: root
        keyPath: ~/.ssh/bastion-key.pem
        port: 22

Options#

      --drain-grace-period-minutes float   Period of time given to each node's pod to terminate gracefully. Timeout should be less than drain-timeout-minutes. If negative, the default value specified in the pod will be used. (default 1.5)
      --drain-timeout-minutes int          Specify how long to wait for drain node. Timeout should be less than total timeout for operation. If 0, then wait indefinitely. (default 3)
  -h, --help                               help for remove

Options inherited from parent commands#

      --bastion-ssh-key string           Specify the ssh key file for bastion host to be used to connect to existing cluster's controller host
                                         (when MKEConfig specifies one for that controller). Controller is picked from MKEConfig as first entry 
                                         with configured SSH. Same precedence and scope as --ssh-key. Has no effect when the chosen controller
                                         has no bastion configured in MKEConfig.
                                         Does NOT override the per-worker bastion key in hosts.yaml.
      --collect-facts-timeout duration   Timeout for the collect-facts phase (gathering per-host facts such as cgroup version over SSH) (default 5m0s)
      --concurrency int                  Maximum number of nodes to configure in parallel, set to 0 for unlimited (default 30)
      --dry-run                          Simulate the node operations without making any changes
  -f, --file string                      path to the config file. Not needed for restore where config is loaded from the backup (default "mke4.yaml")
      --force                            Bypass user confirmations. This will apply changes without any confirmations
      --hosts-path string                path to the hosts config file (default "hosts.yaml")
      --kubeconfig string                path to the kubeconfig file of the existing or restored cluster. DO NOT USE when creating a new cluster (default "/home/runner/.mke/mke.kubeconf")
      --lease-adopt-stale                if true, matching stale lease would be adopted instead of failing with error
      --lease-renew-interval duration    lease renew interval for cluster-wide/node operations, should be < leaseTTL (default 20s)
      --lease-ttl duration               lease TTL for cluster-wide/node operations (default 1m0s)
  -l, --logLevel string                  specify a log level (info, warn, debug, trace, error) (default "info")
      --no-color                         when provided, disables colored output
      --no-pretty-print                  when provided, disables pretty print output formatting
      --ssh-key string                   Specify the ssh key file to be used to connect to existing cluster's controller host. Controller is picked from 
                                         MKEConfig as first entry with configured SSH. If specified, overrides the keyPath stored in MKEConfig 
                                         for that controller; the file must exist locally. Required when MKEConfig's controller keyPath is empty
                                         or not present on this machine.
                                         Does NOT affect SSH keys for the new workers being added — those come from hosts.yaml only.
      --timeout-minutes int              Use specified timeout (in minutes) for node operations. If 0, then wait indefinitely. (default 60)
      --version version[=true]           --version, --version=raw prints version information and quits; --version=vX.Y.Z... sets the reported version