Skip to content

8. A member outside the operation's scope was lost or degraded#

Symptom. A post-check reports a member that "was present and healthy when this operation started" and is now gone or unhealthy, and states it "was never a target of this operation". Raised by both mkectl node add and mkectl node remove.

What it means. VerifyEtcdState records the healthy non-target members as a baseline on its pre-check pass, and the post-check compares against it. A member in that baseline changing state is collateral damage: the operation disturbed a member it was not supposed to touch, or that member failed on its own while the run was in flight.

This is the one case where the first question is quorum, not the member. The operation also changed the member count, so the fault tolerance you had at the start is not the fault tolerance you have now.

Diagnose, in this order.

# 1. Quorum first. Count voters and compare against the pre-operation count.
sudo k0s etcd member-list --data-dir=<dataDir>      # on a SURVIVING controller

# 2. Is the cluster still serving?
sudo /usr/local/bin/etcd-maintenance etcdctl -- endpoint health

# 3. Only then, the lost member's own host.
sudo k0s status
sudo journalctl -u k0scontroller --since -30m

Repair.

  • The host is recoverable — fix the local cause and restart k0scontroller. The member rejoins with its existing data; k0s skips the join when <dataDir>/etcd/member/snap/db exists, which is what you want here. Confirm it is back in the raw member list before re-running.
  • The host is unrecoverable — restore the cluster from the backup this flow took. Both mkectl node add and mkectl node remove take one immediately after the etcd pre-checks pass and before they change anything, so it predates the damage.

Do not retry the add or remove, and do not run k0s etcd leave for the lost member, until you know why it left. Retrying moves the quorum bar again on a cluster that has already lost a voter, and a blind leave on a host that still holds etcd data strands it permanently — see the data-dir caveat in §1.