Skip to content

11. A host cannot be tied to an etcd member at all#

Symptom. ResetControllers refuses to remove a host with one of:

  • "has no usable etcd member identity ... neither an etcd peer address nor a node name could be collected for it";
  • "is not among the discovered etcd members, even though it was classified as <verdict> moments earlier: the membership snapshot changed while this phase was running".

Both mean MKE could not decide which etcd member belongs to the host it was asked to remove, and it refuses to guess — a wrong guess removes a healthy voter.

How MKE identifies a member#

Three sources are reconciled into one row per member. None of them is sufficient alone:

Source Read from Carries
raw etcd member list etcd's member/list API on the helper controller member ID, name, peer URLs, client URLs, learner flag
EtcdMember CRs the kube-apiserver, cluster-scoped status.memberID, status.peerAddress, conditions[type=Joined], spec.leave
control-plane Node objects the kube-apiserver, node.k0sproject.io/role=control-plane node name, internal IPs, Ready condition

They are joined in passes, strongest key first — an etcd member is matched to a CR by member ID, then by peer address, then by lowercased name; a Node is matched by internal IP, then by CR name, then by lowercased member name. Separately, the host being operated on is turned into an identity from the k0s facts CollectFacts gathered over SSH: its etcd peer address and its node name. That identity is what gets matched against the joined rows.

Identification fails when that identity is empty, or when it matches nothing.

Diagnose#

Look at all three sources side by side — this is exactly what the phase could not resolve:

sudo k0s etcd member-list --data-dir=<dataDir>          # on a healthy controller
kubectl get etcdmembers -o wide
kubectl get nodes -l node.k0sproject.io/role=control-plane -o wide

Then, on the host being removed:

sudo k0s status                      # is k0s installed and running at all?
sudo cat /etc/k0s/k0s.yaml           # spec.storage.etcd.peerAddress
hostname                             # the name k0s derives the CR name from

Two patterns account for almost all of these:

  • The host is unreachable or k0s is gone from it. CollectFacts could read nothing, so the identity is empty. Everything else in the cluster is consistent.
  • The host answers, but its private address changed since it joined — a re-IP, a new NIC, a re-image that reused the name. The peer address in etcd no longer matches the host's current one, and the Node's internal IP may not either, so no pass finds a match.

Repair#

  • Host unreachable, but you want to keep it — restore SSH access and re-run. Nothing was changed; both flows are idempotent.
  • Host is gone for good — remove it from the cluster by hand, in this order, then re-run:
# 1. Find its member in etcd by PEER ADDRESS, not by name
sudo k0s etcd member-list --data-dir=<dataDir>

# 2. Remove that member (read the data-dir caveat in §1 first)
sudo k0s etcd leave --peer-address <its-peer-address>

# 3. Clean the two Kubernetes objects it left behind
kubectl delete etcdmember <lowercased-node-name>
kubectl delete node <node-name>

Then remove the host from spec.hosts in the MkeConfig. - The snapshot changed mid-run (the second symptom) — something removed the member out of band while mkectl was working. Simply re-run mkectl node remove; it re-reads all three sources. - The host's address changed — do not try to remove it under its old identity. Correct the host's entry in the MkeConfig to its current address first, so CollectFacts produces an identity that matches what etcd holds.

Do not run k0s etcd leave against a member you have not positively identified. If the identification is wrong, the member you remove is a healthy voter and you have lowered quorum for nothing. When in doubt, match on the peer URL host part — it is the one key all three sources derive from the same private address.

If the mismatch is between the etcd member name and the CR name and they differ only in case, this is not an identification failure — see §5.