Skip to content

9. Cluster not operational after a leave (partially completed removal)#

Symptom. ResetControllers fails with etcd state is not operational after removing host <host> — or with discovering cluster state after removing host <host> when etcd stopped answering the probes outright, which is §6's second shape. Either way the finding notes that the host's removal has already committed and that remaining hosts in the run were not processed.

What it means. This is §6's condition at the worst possible moment. ResetControllers re-checks cluster health after every host it removes, precisely so a multi-host removal stops instead of walking the cluster down below quorum one host at a time. The stop worked — but the cluster is now in a partially-scaled state:

  • the named host is gone from etcd and its k0s is stopped; that is durable and will not roll back;
  • hosts later in the same run were not processed at all;
  • the MkeConfig has not been updated yet — UpdateHostsInMKEConfig runs later in the flow — so MKE still believes the removed host is a member.

Repair.

  1. Get the cluster operational first. Work the signal named in the finding using the table in §6. Do not start another removal until it passes.
  2. Then re-run the same mkectl node remove command, unchanged. It is idempotent: the host already removed is classified clean or left (CR awaiting cleanup) and skipped, the hosts never processed are removed normally, and the config update and CR cleanup run at the end for all of them.

Do not remove the already-processed host from the command to "resume where it stopped" — leaving it out means its MkeConfig entry, its Node object and its EtcdMember CR are never cleaned up.

If quorum cannot be restored, restore from the backup taken at the start of the flow. The removal that committed is in etcd's raft log, so it survives a restart of the survivors — a backup restore is the only way back to the pre-operation membership.