Runbook: etcd membership control-plane#

Some non-happy path in the control-plane mkectl node add / mkectl node remove flow hard-fails with a finding that looks like this:

  Etcd member: {id=2, peerAddress=10.0.1.6, nodeName=cp-1}
  Summary:     ...what is wrong, with the concrete values...
  Remediation: ...the short, immediate instruction...
  Runbook:     https://docs.mirantis.com/mke4/latest/ (topic: etcd-member-stranded)

Start from the topic:. It is a stable identifier for a class of situation, and the topic index below maps it to whichever section documents that class today. Remediation carries only the immediate instruction; everything conditional or multi-step lives here.

Key phases of control plane node add/remove scenarios#

VerifyEtcdState never mutates etcd — repair is a deliberate operator action, because the wrong branch is unrecoverable (see the data-dir caveat under Stranded etcd member).

ResetControllers is the one phase that does mutate etcd, and its order matters for everything below. On a removal it issues member/remove against the helper controller's etcd — by member ID, while the target is still a live voter — then stops k0s on the target, and only then sets spec.leave on the target's EtcdMember CR and waits for it. Several failure modes that a CR-only removal used to produce can therefore no longer arise from mkectl itself; each is marked where that applies.

Topic index#

Topics are the contract between mkectl and this document. The code emits a topic and never a section number, so sections here may be renamed, split, merged or reordered freely — only this table has to follow. Do not change a topic string to match a heading; change the row instead.

`topic:`	Situation, described	Documentation
[`etcd-cluster-not-operational`].	Etcd itself cannot serve: quorum, leader, alarms, backend quota.	§6
[`etcd-member-stranded` ]	Etcd counts a member that nothing owns.	§1
[`etcd-node-without-member`]	A registered control plane that is not a healthy etcd member, including a join that never completed.	§2
[`etcd-stale-member-cr`]	An `EtcdMember` CR that outlived the member it describes.	§3
[`etcd-leave-wedged` ]	A requested leave that cannot complete on its own.	§4
[`etcd-leave-reported-but-member-present`]	A leave reported success while etcd still counts the member.	§5
[`etcd-member-cr-status-unusable`]	The CR status cannot be trusted to describe the member.	§7
[`etcd-member-lost-during-operation` ]	A member outside the operation scope changed state while it ran.	§8
[`etcd-removal-partially-completed`]	A removal committed for some hosts and then stopped.	§9
[`etcd-member-unclassified`]	A member that neither source of truth can classify.	§10
[`etcd-member-identification-failed`]	A host cannot be tied to an etcd member at all: the three sources could not be joined for it.	§11

Topics live in Topic in cli/pkg/etcd/ops/verdict_finding.go. Adding one there means adding a row here; renaming one breaks anybody who bookmarked it, so prefer adding a new topic and retiring the old.

Etcd membership: state vs. intent, and how to read each source correctly#

MKE 4 control planes run stacked etcd: one etcd member per controller+worker node. Two sources describe that membership, they answer different questions, and both must be consulted:

raw etcd member list — the state: what etcd itself counts toward quorum. Run on a healthy controller, with the cluster's actual data dir (never assume /var/lib/k0s — it is user-configurable via MkeConfig's spec.k8sProviderSpecs.dataDir):

sudo k0s etcd member-list --data-dir=<dataDir>

Output is {"members":{"<name>":"<peerURL>"}}. Match members by the host part of the peer URL, not by name: a member that never started has an empty name, and two such members collapse into one entry. The names are still worth reading, for §5.

EtcdMember CRs — cluster-scoped, one per member, name = the node name lowercased. This is the intent and the reconciler's progress channel, not a second copy of the state:

kubectl get etcdmembers -o wide
kubectl get etcdmember <node-name> -o yaml   # spec.leave, status.peerAddress, status.memberID,
                                             # conditions[type=Joined], status.reconcileStatus,
                                             # status.message

status.reconcileStatus: Success does not prove a member was removed/added — see §5. Only the raw list confirms that.

For triage only, one extra read gives per-member etcd liveness that neither source above carries. On any controller:

sudo curl -f -s --max-time 15 http://localhost:18088/cluster/status

That is the etcd-maintenance-service. unreachableMembers is the useful field, but read it precisely: it lists the control-plane node IPs whose etcd-maintenance-service did not answer its Status RPC. That covers a down or wedged etcd, and it equally covers a healthy etcd whose EMS agent is down — the two are not distinguished, so treat an entry as "unconfirmed", not as "etcd is broken here". Caveats: it enumerates control-plane Nodes, so a member with no Node never appears; alarms may already have been auto-disarmed by MKE's own quota bumper, so an empty alarms does not mean the cluster has not been hitting its DB quota; and the service is absent on MKE 4.1.1 and older and unreachable on MKE4k child clusters.

Do not use sudo /usr/local/bin/etcd-maintenance etcdctl -- endpoint health --cluster to judge cluster health: every k0s member advertises its client API on https://127.0.0.1:2379, so that command probes the local node N times and always reports a healthy cluster.

Runbook: etcd membership control-plane#

Key phases of control plane node add/remove scenarios#

Topic index#

Etcd membership: state vs. intent, and how to read each source correctly#

Sections#