Skip to content

5. Leave reports Success but the member is still in etcd (mixed-case hostname)#

Symptom. VerifyEtcdState reports the verdict corrupted removal (VerdictCorruptedRemoval): etcd still counts the member, its CR reads Joined=False with status.reconcileStatus: Success, and spec.leave is still true — often with a status.message left over from an earlier step. Retrying does not help, and no amount of waiting will.

mkectl's own removal no longer produces this. ResetControllers removes the member by ID before it ever writes spec.leave, so by the time k0s's reconciler runs its name lookup the member really is gone and the lookup misses for the right reason. You reach this state from a spec.leave set outside mkectl, from an MKE build older than the ID-first removal, or — most likely — from VerifyEtcdState's pre-emptive member-name check firing before anything is wedged at all (see Prevention below).

Recognise it in one command. The etcd member name and the CR name differ only in case:

sudo k0s etcd member-list --data-dir=<dataDir>   # look at the member NAME, e.g. "Node-1"
kubectl get etcdmembers                          # the CR is "node-1"

What it means. k0s derives two names from the same hostname and lowercases only one of them: etcd's member name is the raw hostname, while the EtcdMember CR name is lowercased (it has to be — Kubernetes object names cannot contain uppercase). The reconciler then looks the member up by name in etcd's member list. On a mixed-case hostname that lookup misses a live, voting member, so the reconciler concludes it has already left, writes Joined=False + reconcileStatus=Success, and returns without ever calling MemberRemove. Because it reports success, its resync never retries.

This is an upstream k0s defect, not an MKE one, and it is why MKE checks the raw member list at all: the CR alone would say the removal worked. Take it seriously — an unnoticed instance leaves a permanently unreachable voting member after the host is reset, which raises the quorum requirement by one and makes the cluster read-only on the next single node failure.

Which k0s versions are affected. Fixed by k0sproject/k0s#7644, which matches CRs to members by member ID instead of by name. Each release line has its own boundary:

  • 1.35 and below — fixed from v1.35.5+k0s.0 (the backport). Not carried further back: v1.34.10+k0s.0 and v1.33.13+k0s.1 still do the name lookup, so anything below 1.35.5 in those lines is affected.
  • 1.36 — fixed from v1.36.1+k0s.0. v1.36.0 shipped before the line picked the fix up, so it is affected however it is spelled (+k0s.1, prereleases).

On a fixed k0s the two names still differ — k0s etcd member-list says Node-1, kubectl get etcdmembers says node-1 — and that is cosmetic, not a wedge.

Prevention. VerifyEtcdState hard-fails the run before an add and before a remove when any member's etcd name and CR name differ byte for byte and the k0s version MKE pins is below its line's boundary (checkMemberNameCase / k0sMatchesEtcdMembersByName, cli/pkg/phase/etcd/verify_etcd_state.go). The check is cluster-wide, not scoped to this operation's targets.

MKE currently pins a k0s below that boundary. On a cluster with even one mixed-case control-plane hostname this gate therefore blocks every mkectl node add and every mkectl node remove — including the removal you would otherwise use to re-join the host under a lowercase name. If you reached this section from that error, no member is wedged yet: upgrade k0s to a version at or above its line's boundary, or use the manual sequence below to take the host out and re-join it lowercase.

Repair. The CR path provably cannot remove this member, so remove it directly. Record the member count before and after — that is the check that matters here.

Do not "fix" this by restarting k0scontroller on the target. It looks like it works and it does not. A k0s start runs createMemberObject, which rewrites spec.leave = false and Joined=True — so the CR stops contradicting etcd, the verdict clears, and the member is still there. You have cancelled the pending leave and hidden the symptom. The member has to be removed from etcd directly, as below.

Remove the member first, and only then stop k0s on the target. The order is not a preference. etcd commits a membership change by majority of the current configuration, so it must be issued while the target is still a live voter. With exactly two voters, stopping the target first leaves a quorum of one out of a required two: the removal can never commit, the survivor steps down, and the cluster answers etcdserver: no leader until the target's k0s is started again. Two voters is reached on the way down from three or four as well, not only on a 2-CP cluster.

# 1. Before: note how many members there are and what quorum is
sudo k0s etcd member-list --data-dir=<dataDir>

# 2. Remove the member, from a HEALTHY controller, while the target is still running
sudo k0s etcd leave --peer-address <target-peer-address>

# 3. After: confirm the peer address is gone and the count dropped by exactly one
sudo k0s etcd member-list --data-dir=<dataDir>

# 4. Now stop k0s on the target — otherwise it keeps restarting an etcd that is no longer
#    a member. Skip if the host is already reset or gone.
sudo systemctl stop k0scontroller     # on the TARGET host

# 5. Clean up the CR, which is now genuinely stale
kubectl delete etcdmember <lowercased-node-name>

Between steps 2 and 4 the target crash-loops: its etcd sees itself removed, exits, and the k0s supervisor respawns it every 5 s. That is noisy and harmless — peers reject raft traffic from a removed ID (410 Gone, removed member) and the removed set is persisted, so it cannot rejoin or influence the survivors. It also cannot re-add itself: k0s skips the join whenever <dataDir>/etcd/member/snap/db exists. Do not wipe the target's data dir before step 4.

The data-dir caveat in §1 does not block you here: this host is being removed deliberately and is reset afterwards, so losing its etcd data is the intent. Do check that you are running step 2 against the target's peer address and not a survivor's.

Then re-run mkectl node remove --scope=controller+worker to finish the host reset and the config update.

Avoiding it entirely. Use lowercase hostnames on control-plane nodes. This is worth checking before an MKE3→MKE4 upgrade in particular, where hostnames are inherited rather than chosen: a mixed-case control plane can be added successfully and only becomes unremovable later.