etcd member verdicts#

When you add or remove control plane nodes with --scope=controller+worker, mkectl classifies every etcd member it can see into exactly one verdict. This verdict ascertains whether the operation can proceed for that host, and it is the value quoted in the failure output.

Info

Verdicts apply to control plane operations only. A worker operation does not touch etcd membership and never produces one.

Derive a verdict#

MKE 4 control planes run stacked etcd; one etcd member per controller+worker node. Two sources describe that membership, and they address different questions:

The raw etcd member list, which is read on the helper controller with k0s etcd member-list, is the state: what etcd itself counts toward quorum. Members are matched by the host part of the peer URL, not by name.
The EtcdMember custom resources, one per member, cluster-scoped and named after the lowercased node name, are the intent and the k0s progress channel for the reconciler.

A verdict is the combination of raw etcd member entry and the EtcdMember custom resource. mkectl never derives a verdict from one source alone.

Raw etcd member list	`EtcdMember` CR	`Joined` condition	`spec.leave`	Leave annotation matches `status.memberID`	Verdict
present	present	missing (`Unknown`)	any	not analyzed	`CR status never written`
present	present	`True`	`false`	not analyzed	`healthy`
present	present	`True`	`true`	not analyzed	`leave requested`
present	present	`False`	`false`	not analyzed	`CR status stale`
present	present	`False`	`true`	not analyzed	`corrupted removal`
absent	present	missing (`Unknown`)	any	not analyzed	`CR status never written`
absent	present	`False`	any	not analyzed	`left (CR awaiting cleanup)`
absent	present	`True`	`true`	not analyzed	`left etcd, CR isn't updated`
absent	present	`True`	`false`	matches	`left etcd, CR isn't updated`
absent	present	`True`	`false`	empty or mismatched	`oob etcd member removal, stale EtcdMember CR`
absent	absent	—	—	not analyzed	`clean`
present	absent	—	—	not analyzed	`hazard: EtcdMember CR missing`
anything else	anything else	any	any	not analyzed	`unknown`

The leave annotation is mke.mirantis.com/leave-requested-by-mkectl, which mkectl writes when it initiates a removal. It is what distinguishes a removal that MKE itself performed from one performed out of band.

Note

A Joined=True condition reports join state, not etcd liveness. A member whose etcd process is wedged still reports Joined=True, and thus still yields the healthy verdict. Node readiness and k0s health are checked separately, earlier in the same command.

Verdict reference#

Verdict	Meaning	Runbook topic
`healthy`	etcd considers the member and its CR reports to be a finished join. The normal state of a working member.	—
`clean`	Neither source knows the peer address. It is not a member. The normal state of a host that has never joined, or that was fully removed.	—
`left (CR awaiting cleanup)`	The member left etcd and its CR is awaiting cleanup. This is the expected post-state of a successful removal; `mkectl` deletes the CR itself.	`etcd-stale-member-cr`
`leave requested`	A leave was requested through the CR and the member is still in etcd. Transient during a removal that MKE is performing.	`etcd-leave-wedged`
`left etcd, CR isn't updated`	The removal succeeded, but the write that would have updated the CR status was lost. The member is gone.	`etcd-stale-member-cr`
`oob etcd member removal, stale EtcdMember CR`	etcd does not count the member, but the CR still claims `Joined=True` and carries no matching leave annotation. The member was removed out of band and a stale CR was left behind.	`etcd-stale-member-cr`
`hazard: EtcdMember CR missing`	etcd counts a member for quorum that no CR owns, because either the join never finished or the CR was deleted before the leave was completed. With the CR gone, the CR-driven removal path can no longer remove it.	`etcd-member-stranded`
`corrupted removal`	The CR reports a completed leave while etcd still counts the member, so the CR-driven removal reported success without removing anything.	`etcd-leave-reported-but-member-present`
`CR status never written`	The CR has no `Joined` condition at all, and thus the member state cannot be determined from it.	`etcd-member-cr-status-unusable`
`CR status stale`	The CR status contradicts etcd in a way that cannot be reconciled, and thus the member state cannot be trusted.	`etcd-member-cr-status-unusable`
`unknown`	Neither source of truth can classify the member. Normally a sign of an abnormal cluster state.	`etcd-member-unclassified`

Warning

corrupted removal is the one verdict where the otherwise standard repair — restarting k0scontroller on the affected host — is harmful. A k0s start rewrites spec.leave and the Joined condition, which clears the verdict without removing the member and cancels the pending leave. In this case, adhere to the runbook topic.

Verdicts by operation#

The same verdict can be perfectly normal in one operation and a failure in another. mkectl evaluates each host in scope against the operation it is about to perform.

Verdict	Host being added for the first time	Host already a control plane node, added again	Host being removed
`healthy`	Rejected. A member on a host that is not part of the cluster is a stranded member.	Accepted	Accepted
`clean`	Accepted	Rejected — a control plane node that contributes no etcd vote	Accepted
`left (CR awaiting cleanup)`	Accepted; the stale CR is deleted first.	Rejected	Accepted
`left etcd, CR isn't updated`	Rejected	Rejected	Accepted; the host is excluded from the etcd removal step.
`leave requested`	Rejected	Rejected	Accepted; `mkectl` waits for the leave to complete.
Every other verdict	Rejected	Rejected	Rejected

In addition, and independent of the above table:

Every etcd member outside the operation scope must be healthy, both before and after the operation. An addition joins the new node as a voting member, which raises the quorum bar before the newcomer can contribute, and thus holding quorum at that time is not sufficient.
After an addition, every new host must have reached healthy.
After a removal, every target must be gone from etcd, or at worst report left (CR awaiting cleanup) or left etcd, CR isn't updated, which mkectl then cleans up.

Read a verdict failure#

A rejected verdict is reported as a finding:

  Etcd member: {id=2, peerAddress=10.0.1.6, nodeName=cp-1}
  Summary:     ...what is wrong, with the concrete values...
  Remediation: ...the short, immediate instruction...
  Runbook:     <URL of runbook docs> (topic: etcd-member-stranded)

Start from the topic: value, which is a stable identifier for a class of situation, and the etcd membership runbook maps it to whichever section documents that class. Remediation carries only the immediate instruction; everything conditional or multi-step is in the runbook.

When several hosts are in scope, mkectl reports a finding for each one that was rejected.

Note

mkectl never mutates etcd to repair any of verdict states. Repair is a deliberate operator action, because the wrong repair branch is unrecoverable; for example, k0s skips the join when etcd data already exists on a host, so issuing a leave against a host that still holds data permanently strands it.