Skip to content

2. Node registered but no etcd member#

Symptom. VerifyEtcdState fails with "control-plane node X is registered in the cluster but no etcd member could be matched to it": a Node carrying node.k0sproject.io/role=control-plane that none of the discovered etcd members joined to. Or a host in scope of an add is already a registered controller (scenario 2/3) but its peer address is absent from the raw list, or its CR is missing / Joined=False.

The node-side check is deliberately scoped to control planes outside the operation: a target in this state is reported by the per-intent checks instead, which name the host address you typed rather than a node name you would have to map back to a host.

What it means. A running, registered controller that is not a healthy etcd member. Adding another voting member on top of this raises the quorum bar while a controller that looks live contributes nothing.

Diagnose.

kubectl get nodes -l node.k0sproject.io/role=control-plane
sudo k0s etcd member-list --data-dir=<dataDir>          # on a healthy controller
sudo k0s status                                        # on the suspect controller
sudo journalctl -u k0scontroller --since -30m           # etcd startup errors
kubectl get etcdmember <node-name> -o yaml              # status.message is the reconciler's channel

Three variants reach this topic, and they differ only in what noticed them and when:

  • a host that was already a control plane and whose etcd is not a member — caught on the pre-check of an add or a remove, against a host that is in scope;
  • a host mkectl node add just added whose join never completed — caught on the post-check, with the summary "etcd counts no member for it: the join never completed". The rest of the cluster is untouched in this variant, which is what makes it safe to simply fix and re-run;
  • a control plane that has nothing to do with your operation — caught by the node-side check on every pass of both intents, with the summary quoted at the top of this section. This one was already broken before you ran anything, and the operation is refused because adding or removing a voter on top of a control plane that contributes no vote changes the quorum arithmetic you think you have.

Repair. There is no safe automatic repair; work it cheapest-first and do not start with anything destructive — etcd failing to start is usually a fixable local fault, and a wipe throws away the evidence that says which one.

  1. Find out why etcd never joined or never started. On that host:
sudo k0s status
sudo journalctl -u k0scontroller --since -30m

Look for a peer unreachable on port 2380, a certificate error, or clock skew. For a host that was already a member, look for etcd startup errors instead: disk, certificates, clock skew.

  1. Fix the cause and restart k0s on that host.
sudo systemctl restart k0scontroller

k0s recreates the EtcdMember CR as soon as its local etcd answers and the kube-apiserver is reachable. Confirm with kubectl get etcdmembers and the raw member list.

  1. Re-run the original command. Both mkectl node add and mkectl node remove are idempotent.

  2. Only if the host is unrecoverable, remove it (mkectl node remove --scope=controller+worker) and re-add it, or clean it by hand (k0s stop; k0s reset && rm -rf <dataDir>/* && rm -rf /etc/k0s).

Two things not to do. Do not wipe the host's data dir before step 1 — it is the evidence. And do not run k0s etcd leave for this host: on the join-never-completed variant etcd has no member to remove, and on the already-a-member variant a leave against a populated data dir strands the host permanently (see the caveat in §1).