6. etcd cluster is not operational

Symptom. VerifyEtcdClusterState or VerifyEtcdState fails, in one of two shapes. This is the most common hard fail in both flows.

1. The probes answered and the answers are bad — etcd state is not operational, followed by a finding naming the signals that failed and a full state report. That is the table below.
2. The probes did not answer — discovering etcd state, followed by a finding whose summary reads "the etcd cluster state could not be read from <host>" and whose details lead with probe failures:. No signal is named because none was measured; the probe errors are the diagnosis. This is the shape an etcd that is down or unreachable on the helper controller produces, and it is the more likely of the two: the state read returns the moment any probe fails, so the signal table is never evaluated. Go straight to k0s status and journalctl -u k0scontroller on the host named in the summary — it is that host's etcd, not necessarily the cluster, that failed to answer.

Both flows run this check more than once. On the pre-check passes — the standalone VerifyEtcdClusterState phase, and again inside the first VerifyEtcdState — it fires before any member is classified and before any host is touched, so there is nothing to undo. On the post-check pass it fires after the hosts were added or removed; if that is where you are, the state named in §8 applies as well. The failure raised by ResetControllers between two removals is §9, and its message says so explicitly.

What it means. Operational() is an AND over independent signals, each read from the helper controller's own etcd. More than one can be at fault at once, and the finding names every one of them:

Signal in the finding	What it means	What to do
/readyz does not pass	etcd's own composite readiness check failed — it covers data corruption, serializable read, linearizable read and non-learner	read the reason in the finding; it names the failing sub-check, and that sub-check is the real case
/readyz/linearizable_read does not pass	the direct proof that quorum is not operational: the read goes through raft's ReadIndex, which needs a leader and a majority	bring every control-plane host back up, confirm k0scontroller is running on each, then re-run
etcd knows no leader	an election is in progress, or quorum is lost	no membership change can commit without a leader; restore enough voting members to form a quorum first
quorum is not confirmed (N voting member(s), quorum size M)	fewer live voters than the majority the current configuration needs	the count is the diagnosis: N voters against quorum size M. Recover voters until N >= M
the NOSPACE alarm is raised	the backend hit its quota; etcd is read-only	defragment through the etcd-maintenance service or raise the etcd storage quota, then disarm the alarm
the CORRUPT alarm is raised	etcd detected data corruption	not recoverable by scaling — restore the cluster from a backup before adding or removing anything
the etcd backend on <host> is at N% of its quota	≥ 80% used, i.e. close to NOSPACE. Read from that one host, so it is one member's view of the backend	a backend that reaches its quota mid-operation leaves the control plane half-scaled. Defragment or raise the quota before retrying

Diagnose. The finding already carries the full report. To read it live:

# on the helper controller — the same source the phase uses
sudo curl -f -s --max-time 15 http://localhost:18088/cluster/status   # per-member view
sudo k0s etcd member-list --data-dir=<dataDir>                        # who is still a voter

Read the unreachableMembers caveat in the Background section of the runbook overview before acting on the first command: an entry there means the etcd-maintenance service did not answer, which is not the same as etcd being down.

Repair. Resolve the named signal, then re-run. Both mkectl node add and mkectl node remove are idempotent and nothing was changed by the failed run, so there is no cleanup to do first.

The backend-usage threshold deserves a note: it is MKE's own guard at 80%, not an etcd alarm. etcd is still fully functional at that point. The check exists because a control-plane scale-up writes to etcd while raising the quorum bar, and hitting NOSPACE halfway through is far worse than being told to defragment first.