Skip to content

3. EtcdMember CR without an etcd member (stale CR)#

Symptom. etcd does not count the member, but its CR still reports Joined=True. Two different verdicts land here, and only the second is an anomaly:

  • left etcd, CR isn't updated (VerdictLeftCRNotUpdated) — MKE's own removal succeeded and the CR's status write was lost. Self-clearing; see below.
  • oob etcd member removal, stale EtcdMember CR (VerdictOOBRemovalStaleCR) — the real stale-CR anomaly. Continue below.

The discriminator is spec.leave plus MKE's own annotation, not spec.leave alone. MKE stamps mke.mirantis.com/leave-requested-by-mkectl with the member's hex ID before it removes anything, so a CR carrying that stamp for this member's ID is MKE's own removal even when spec.leave has gone back to false — the target's k0s clears spec.leave on every start:

kubectl get etcdmember <node-name> \
  -o jsonpath='leave={.spec.leave} memberID={.status.memberID} stamp={.metadata.annotations.mke\.mirantis\.com/leave-requested-by-mkectl}{"\n"}'

The stamp is <hexID>@<timestamp>; compare only the part before the @ against memberID.

  • spec.leave: true, or the annotation's hex ID equals status.memberID — this is VerdictLeftCRNotUpdated, not an anomaly, and it needs no manual repair. mkectl node remove runs CleanupStaleEtcdMemberCRs a second time at the end of the flow with CRs-with-missing-members in scope, and that pass deletes exactly this CR. Re-run the removal rather than deleting by hand.

If a removal is still in flight and the CR looks stuck, the reconciler has no periodic tick — it wakes only on an EtcdMember write, a leader change, or its 10s/30s retry timers. Re-running mkectl node remove nudges it, since it re-asserts spec.leave. Failing that:

kubectl annotate etcdmember <node-name> mke.mirantis.com/nudge="$(date +%s)" --overwrite
  • spec.leave: false and no matching annotation — this is VerdictOOBRemovalStaleCR. Continue below.

What it means (VerdictOOBRemovalStaleCR). The member was removed out of band (someone ran k0s etcd leave or etcdctl member remove directly), so etcd no longer counts it, but the CR was never updated. CleanupStaleEtcdMemberCRs deletes only the verdicts in its target set — VerdictMemberLeftCRNeedsCleanup, plus VerdictLeftCRNotUpdated on the post-removal pass — and VerdictOOBRemovalStaleCR is in neither, so the object would accumulate silently.

Repair. Confirm the member really is gone from etcd, then delete the stale object:

sudo k0s etcd member-list --data-dir=<dataDir>   # the peer address must be absent
kubectl get etcdmember <node-name> -o yaml       # confirm this is the right object
kubectl delete etcdmember <node-name>

The CR is not recreated behind your back: k0s only creates it from a controller that can resolve its own member ID from its local etcd, which a removed member cannot.

If the corresponding Node still exists and the host is still running k0s, you have case §2 as well — resolve that first.