Skip to content

4. Wedged leave (spec.leave never completes)#

Symptom. ResetControllers or WaitForEtcdMembers times out, with a finding per stuck member reading "the leave … never completed: nothing wrote spec.leave=true with Joined=False for it". Because ResetControllers removed the member from etcd before requesting the leave, the raw member list is normally already satisfied and only the CR half is stuck.

You can also land here from the VerifyEtcdState pre- or post-check of a later node add or node remove, with a finding reading "for etcd member leave is requested via EtcdMember CR" or one whose summary ends in "(leave requested)": a member outside that operation's scope, or an add target, still carries spec.leave=true with Joined=True, usually left behind by an earlier removal that was interrupted. The check cannot tell an in-flight leave from a wedged one. Read status.message on the CR first: if it is one of the progress messages below, wait for Joined=False and re-run; if it does not change within a few minutes, treat it as wedged and continue with this section. ResetControllers may instead fail outright with "the leave request for host … keeps being cleared by its own k0s controller" — that is case 3 below, after MKE re-asserted spec.leave three times and gave up. Otherwise status.message on the target's CR reads one of:

  • No k0s controller lease found — the wedge in case 2 below;
  • Member is still active; waiting for it to shutdown — normal for a live target, and bounded;
  • The only active controller cannot leave the etcd cluster — the target is the last controller k0s can see. Bring another controller online, or use mkectl reset if you meant to tear the cluster down;
  • Waiting for another controller to take over with an empty reconcileStatus — progress, not failure: the target was the lease leader, yielded the k0s-endpoint-reconciler lease, and a surviving controller finishes the removal within seconds.

What it means. The EtcdMemberReconciler runs on the lease leader only and is driven entirely by the CR. It refuses to remove a member while it believes the target's k0s is still running, which it decides from the target's own heartbeat Lease k0s-ctrl-<node-name> in kube-node-lease. Three ways this stalls:

  1. no lease leader — nothing reconciles at all;
  2. the k0s-ctrl-<node-name> lease object is missing (not merely expired) while the CR still carries the k0s.k0sproject.io/shutdown-on-leave label: the reconciler reports No k0s controller lease found and returns without removing the member, forever. An expired lease is fine — it falls through to the removal.

Not reachable from mkectl itself. The reconciler only gets as far as the lease check when the member is still in etcd's member list; ResetControllers has already removed it by then, so the reconciler takes its "member marked for leave but not in the member list" branch and finishes. You reach this wedge from a spec.leave set outside mkectl, or from an MKE build older than the ID-first removal; 3. the target's k0s restarted — every start sets spec.leave = false and drops the shutdown label, cancelling the pending leave. ResetControllers re-asserts the leave up to three times before failing with the message quoted above, so a single restart is absorbed and a restart loop is not.

Diagnose.

kubectl get etcdmember <node-name> -o yaml
kubectl -n kube-node-lease get lease k0s-ctrl-<node-name> -o yaml   # missing? that is the wedge
kubectl -n kube-node-lease get lease k0s-endpoint-reconciler -o yaml # holderIdentity == lease leader
kubectl get etcdmember <node-name> \
  -o jsonpath='{.metadata.labels.k0s\.k0sproject\.io/shutdown-on-leave}{"\n"}'

Repair.

  • No lease leader — restore at least one healthy controller running k0s; the new leader's resync walks every CR and finishes the job.
  • Target's k0s restarted — re-run mkectl node remove --scope=controller+worker; the leave is level-triggered, so it is re-asserted. Do not restart the target again until the member is gone.
  • Missing k0s-ctrl-<node-name> lease — remove the member directly from a healthy controller, then clean the CR:
sudo k0s etcd leave --peer-address <target-peer-address>
sudo k0s etcd member-list --data-dir=<dataDir>   # confirm absent
kubectl delete etcdmember <node-name>

Read the data-dir caveat in §1 first — here it is normally satisfied, because the target is being removed and its host is reset afterwards.

Never delete an EtcdMember CR whose spec.leave is still unsatisfied while the member is present in the raw list: the reconciler has no finalizers and no delete path, so the CR-driven removal is gone with it and you are left with the k0s etcd leave --peer-address route above.

If the CR instead says the leave already succeeded while the member is still in the raw list, this is not a wedge — it is §5.