- Home
- Troubleshooting
- Recover from etcd quorum loss
Recover from etcd quorum loss#
MKE 4 runs stacked etcd, with one etcd member on each controller node. etcd
commits a change only when a majority of its voting members, the quorum, agrees
to it. A cluster of N controllers therefore tolerates the loss of (N-1)/2
controllers: one of three, or two of five. When more controllers than that are
lost, etcd stops serving requests and the Kubernetes API goes down with it.
Workloads on the worker nodes keep running and serving traffic, but you cannot
schedule, scale, or change anything in the cluster.
If you can bring the lost controllers back online with their data intact, do so. etcd regains quorum on its own once enough members return, and no further action is required. If the controllers are lost permanently, you can recover the cluster in place, without a full restore, provided that at least one controller survives with its etcd data intact. You force the surviving controller to start a new single-member etcd cluster that keeps all of its data, you remove the lost controllers from the cluster, and then you add new controllers to re-establish high availability.
| Situation | Recovery |
|---|---|
| The lost controllers can be brought back online. | Bring them back. etcd regains quorum without intervention. |
| A majority of the controllers is lost permanently, and at least one controller survives with intact etcd data. | Follow the procedure on this page. |
No controller survives, or etcd raised the CORRUPT alarm. |
Restore the cluster from a backup. |
Warning
-
Use this procedure only as a last resort. Forcing a new etcd cluster rewrites the etcd membership, and the change cannot be undone once new controllers join.
-
The recovered cluster contains only the data that the surviving controller had committed. Any writes that reached only the lost controllers are lost.
Symptoms#
kubectlcommands time out or fail to connect, and the MKE 4 Dashboard is unavailable.-
On a surviving controller, the etcd endpoint status reports
etcdserver: no leaderin theerrorssection:sudo curl --cacert /var/lib/k0s/pki/etcd/ca.crt --cert /var/lib/k0s/pki/etcd/server.crt --key /var/lib/k0s/pki/etcd/server.key -s https://127.0.0.1:2379/v3/maintenance/status -X POST | jqExample output:
{ "header": { "cluster_id": "7346508586475956732", "member_id": "11132754253862912738", "revision": "3878982", "raft_term": "180" }, "version": "3.6.11", "dbSize": "98205696", "raftIndex": "4465940", "raftTerm": "180", "raftAppliedIndex": "4465940", "errors": [ "etcdserver: no leader" ], "dbSizeInUse": "27922432", "storageVersion": "3.6.0", "dbSizeQuota": "2147483648", "downgradeInfo": {} }Note
The commands in this topic use the default k0s data directory,
/var/lib/k0s. If you set a differentdataDir, substitute it, as described in Before you begin.
Before you begin#
Verify that you have the following in place:
- SSH access with root privileges to every surviving controller node.
- The location of the k0s data directory. The procedure uses the default,
/var/lib/k0s. If you setspec.k8sProviderSpecs.dataDirin theMkeConfigfile, substitute that value wherever/var/lib/k0sappears, and add--data-dir <dataDir>to thek0s kubectlandk0s resetcommands. - The
mkectlbinary that matches the version of the cluster. For more information, refer to Version alignment. - Hosts for the replacement controller nodes that meet the system requirements and have an absent or empty k0s data directory. You can reuse the hosts of the lost controllers once you clean them, as described in Re-establish high availability.
- On the hosts for the replacement controller nodes, firewalld is either disabled or configured for MKE 4.
- The most recent cluster backup, and the
mkectlbinary that created it, in case the in-place recovery fails.
Fence the lost controllers#
An unreachable controller is not necessarily a stopped controller. If the lost controllers are only cut off from the survivor, for example by a network partition, they still form a majority of the original etcd membership. In that case, once you force a new cluster on the survivor, two etcd clusters run independently and their data diverges.
Before you continue, verify that none of the lost controllers can return to the cluster during the procedure. For each lost controller, do one of the following:
-
If you can reach the host, stop k0s and prevent it from starting at boot:
sudo systemctl disable --now k0scontroller -
Otherwise, power it off or disconnect it from the network, and keep it that way until you clean it in Re-establish high availability. Disable any automatic power-on in the hypervisor or the out-of-band management.
Warning
The k0scontroller service is enabled at boot. A controller that you power
off starts k0s as soon as it boots again, with the etcd data and membership
of the original cluster. If enough of the lost controllers run at the same
time to form a majority of the original membership, for example both lost
controllers of a three-controller cluster, they elect a leader and run a
second copy of the cluster on stale data.
Select the surviving controller#
If only one controller survives, it is the surviving controller for the rest of this procedure.
If more than one controller survives, for example two of five, select the one that holds the most recent etcd data:
-
On each surviving controller, read the local etcd status. The command answers even when etcd has no quorum.
sudo curl --cacert /var/lib/k0s/pki/etcd/ca.crt --cert /var/lib/k0s/pki/etcd/server.crt --key /var/lib/k0s/pki/etcd/server.key -s https://127.0.0.1:2379/v3/maintenance/status -X POST | jqExample output:
{ "header": { "cluster_id": "7346508586475956732", "member_id": "11132754253862912738", "revision": "3878982", "raft_term": "180" }, "version": "3.6.11", "dbSize": "98205696", "raftIndex": "4465940", "raftTerm": "180", "raftAppliedIndex": "4465940", "errors": [ "etcdserver: no leader" ], "dbSizeInUse": "27922432", "storageVersion": "3.6.0", "dbSizeQuota": "2147483648", "downgradeInfo": {} } -
Compare the
raftAppliedIndexvalues. The controller with the highest value holds the most recent data. If the values are equal, select any of them. Keep the other surviving controllers as fallback options, and do not reset them yet. If the recovery fails on the selected controller, stop k0s on it, prevent it from starting at boot, and try the recovery on the next surviving controller inraftAppliedIndexorder. -
On every other surviving controller, stop k0s and prevent it from starting at boot:
sudo systemctl disable --now k0scontrollerThese controllers leave etcd when you force the new cluster. Do not reset them yet, as they are your fallback if the recovery fails on the selected controller. After the recovery succeeds, remove their records together with those of the lost controllers, and clean their hosts before you add them back.
Force a new etcd cluster#
Run the following steps on the surviving controller:
-
Stop the etcd maintenance service:
sudo systemctl stop etcdmaintenance.serviceThe service can take several minutes to stop, because it keeps trying to reach the unresponsive etcd.
Verify that it is stopped:
sudo systemctl status etcdmaintenance.serviceExample output:
× etcdmaintenance.service - Tool for performing etcd maintenance tasks. Loaded: loaded (/etc/systemd/system/etcdmaintenance.service; enabled; vendor preset: enabled) Active: failed (Result: timeout) since Fri 2026-11-06 15:27:41 UTC; 29min ago Process: 907 ExecStart=/usr/local/bin/etcd-maintenance run --config=/var/lib/etcd-maintenance/config.yaml (code=killed, signal=KILL) Main PID: 907 (code=killed, signal=KILL) CPU: 36.333s -
Stop k0s:
sudo /usr/local/bin/k0s stopBecause the controller is also a worker node, workloads that run on it can be interrupted until k0s starts again.
Note
The procedure calls k0s by its full path,
/usr/local/bin/k0s. On distributions such as RHEL,/usr/local/binis not in thesecure_pathofsudo, andsudo k0sfails withsudo: k0s: command not found.Verify that k0s is stopped before you continue, as copying the data directory of a running etcd does not produce a usable copy:
sudo systemctl is-active k0scontrollerExample output:
inactiveIf the command prints
active, k0s is still running. Do not continue until it stops. -
Copy the etcd data directory to a location outside the k0s data directory. The copy is your way back to the current state if the recovery fails.
sudo cp -a /var/lib/k0s/etcd <backup-path> -
In the
/etc/k0s/k0s.yamlfile, addforce-new-cluster: "true"underspec.storage.etcd.extraArgs. IfextraArgsis not present, create it. Keep every other setting, including any existingextraArgsentries, unchanged.spec: storage: etcd: extraArgs: force-new-cluster: "true" -
Start k0s:
sudo /usr/local/bin/k0s start -
Verify that etcd runs as a single-member cluster. etcd can take a few minutes to respond after k0s starts.
-
Verify that the status endpoint reports a
leadervalue that matchesheader.member_id, and that the output has noerrorssection: -
Verify that the member list includes only the surviving controller:
-
Verify that the
readyzendpoint returnsok:
-
-
Verify that the Kubernetes API responds:
Example output:sudo /usr/local/bin/k0s kubectl get nodesconsole { .no-copy } NAME STATUS ROLES AGE VERSION controller-1 Ready control-plane 8d v1.35.4+k0s controller-2 NotReady control-plane 22h v1.35.4+k0s controller-3 NotReady control-plane 22h v1.35.4+k0s worker-1 Ready <none> 8d v1.35.4+k0sThe lost controllers can still appear asReadyright after k0s starts. Their status changes toNotReadywithin a few minutes.
!!! note "Note"
If etcd does not start, review the k0s logs with sudo journalctl -u k0scontroller. To return to the state before this step, stop k0s, replace /var/lib/k0s/etcd with the copy from step 3, and remove force-new-cluster from /etc/k0s/k0s.yaml.
Remove the force-new-cluster flag#
Warning
Remove force-new-cluster as soon as etcd runs as a single-member cluster.
etcd applies the flag every time it starts. If you leave the flag in place,
the next restart of k0s, including the restart that mkectl node add
performs on this controller, forces a single-member cluster again and
removes every controller that has joined since.
-
In the
/etc/k0s/k0s.yamlfile, remove theforce-new-clusterentry. If you createdextraArgsin the previous section, remove it as well. The file must match its original content, otherwisemkectl node addrewrites it and restarts k0s on this controller while it adds the new controllers. -
Restart k0s:
sudo /usr/local/bin/k0s stop sudo /usr/local/bin/k0s start -
Verify that etcd is still a healthy single-member cluster, using the same commands as before:
sudo curl --cacert /var/lib/k0s/pki/etcd/ca.crt --cert /var/lib/k0s/pki/etcd/server.crt --key /var/lib/k0s/pki/etcd/server.key -s https://127.0.0.1:2379/v3/maintenance/status -X POST | jq sudo curl --cacert /var/lib/k0s/pki/etcd/ca.crt --cert /var/lib/k0s/pki/etcd/server.crt --key /var/lib/k0s/pki/etcd/server.key -s https://127.0.0.1:2379/readyz sudo /usr/local/bin/k0s kubectl get nodes -
Start the etcd maintenance service:
sudo systemctl start etcdmaintenance.serviceVerify that it started:
sudo systemctl status etcdmaintenance.serviceExample output:
● etcdmaintenance.service - Tool for performing etcd maintenance tasks. Loaded: loaded (/etc/systemd/system/etcdmaintenance.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2026-11-06 15:57:29 UTC; 3min 8s ago Main PID: 366967 (etcd-maintenanc) Tasks: 17 (limit: 28835) Memory: 19.6M CPU: 148ms CGroup: /system.slice/etcdmaintenance.service └─366967 /usr/local/bin/etcd-maintenance run --config=/var/lib/etcd-maintenance/config.yaml -
Verify that the Kubernetes API is available:
sudo /usr/local/bin/k0s kubectl get nodesThe lost controllers are listed as
NotReady. It can take a few minutes after k0s starts for their status to change fromReady.
Remove the lost controllers from the cluster#
The lost controllers no longer belong to etcd, but the cluster still records each of them in several places:
- The
EtcdMemberobject. - The
ControlNodeobject. - The
Nodeobject. - The entry in
spec.hostsof theMkeConfigobject.
You must remove all four manually, because mkectl node add pre-flight checks fail for as long as any of these
records remains.
Note
If more than one controller survived, the other surviving controllers that you stopped in Select the surviving controller also left etcd when you forced the new cluster, and the cluster still records them in the same places. Remove their records in this section as well. This removes only objects from the cluster. Their hosts keep their etcd data until you clean them in Re-establish high availability.
Run the kubectl commands in this section with the kubeconfig of the cluster.
If the external address of the cluster does not route to the surviving
controller, run them on the surviving controller as
sudo /usr/local/bin/k0s kubectl instead.
-
List the etcd members and the
EtcdMemberobjects:sudo curl --cacert /var/lib/k0s/pki/etcd/ca.crt --cert /var/lib/k0s/pki/etcd/server.crt --key /var/lib/k0s/pki/etcd/server.key -s https://127.0.0.1:2379/v3/cluster/member/list -X POST | jqExample output:
{ "header": { "cluster_id": "7346508586475956732", "member_id": "11132754253862912738", "raft_term": "181" }, "members": [ { "ID": "11132754253862912738", "name": "controller-1", "peerURLs": [ "https://100.64.100.5:2380" ], "clientURLs": [ "https://127.0.0.1:2379" ] } ] }kubectl get etcdmembers -o wideExample output:
NAME PEER ADDRESS MEMBER ID JOINED RECONCILE STATUS controller-1 100.64.100.5 9a7f7d029137c2e2 True controller-2 100.64.100.6 f772bd71ba9502d2 True controller-3 100.64.100.7 e1d8ae4806478cb1 TrueThe
EtcdMemberobject of each lost controller has aPEER ADDRESSthat is absent from the etcd memberspeerURLslist. In the example, controller-2 and controller-3 are the lost controllers. -
Delete the
EtcdMemberobject of each lost controller. The object name is the node name in lowercase. Do not delete the object of the surviving controller.kubectl delete etcdmember <lowercase-node-name> -
Delete the
ControlNodeobject of each lost controller. The object name is the hostname of the controller, which is normally the same as the node name. Do not delete the object of the surviving controller.kubectl get controlnodes kubectl delete controlnode <node-name> -
Delete the
Nodeobject of each lost controller:kubectl delete node <node-name>Kubernetes then deletes the pods that are bound to the lost nodes, and their controllers recreate them on the remaining nodes.
-
Wait for the MKE operator and the
MkeReleaseobject to become ready.The MKE operator validates every change to the
MkeConfigobject, and rejects it until theMkeReleaseobject of the cluster version isReady. Once you delete theNodeobjects, the pods that ran on the lost controllers, which can include the MKE operator and the k0rdentsource-controller, restart on the remaining nodes. Until they do, theMkeReleaseobject can reportREADYasFalsefor several minutes.Wait for the
mke-operatorpod in themkenamespace to becomeReady:kubectl -n mke get podsWait for the
MkeReleaseobject to reportREADYasTrue:kubectl get mkereleases -
Remove the lost controllers from the
MkeConfigobject:kubectl -n mke edit mkeconfig mkeIn
spec.hosts, delete the entry of each lost controller, keep the entries of the surviving controller and of the worker nodes, and save the object. Keep a copy of the deleted entries if you intend to reuse their SSH settings in the hosts file for the new controllers.Note
If the edit is rejected because the validation webhook is not available, or with the error
MkeRelease for specified MKE version is not yet ready, repeat the previous step, and then edit the object again.
Re-establish high availability#
-
Prepare the hosts for the new controllers.
To reuse the host of a lost controller, or of a surviving controller that you stopped while selecting the surviving controller, first clean it. Such a host still holds the etcd data and membership of the original cluster, and unless you disabled k0s on it, k0s starts as soon as the host boots.
Warning
Bring the hosts back one at a time. Boot one host, and stop and disable k0s on it before you boot the next one. If enough of these hosts run k0s at the same time to form a majority of the original membership, they run a second copy of the cluster on stale data. For more information, refer to Fence the lost controllers.
-
As soon as the host boots, stop k0s and prevent it from starting at boot:
sudo systemctl disable --now k0scontroller -
Remove the etcd maintenance service:
sudo systemctl stop etcdmaintenance.service sudo systemctl disable etcdmaintenance.service sudo rm -rf /etc/systemd/system/etcdmaintenance.service sudo systemctl daemon-reload sudo /usr/local/bin/etcd-maintenance uninstall sudo rm -f /usr/local/bin/etcd-maintenance -
Reset k0s, passing the kubelet root directory of the cluster (
spec.kubelet.kubeletRootDirin theMkeConfigobject,/var/lib/kubeletby default), and remove what remains of the k0s data directory and the k0s configuration:sudo /usr/local/bin/k0s reset --kubelet-root-dir <kubelet-root-dir> if [ -d /var/lib/k0s ]; then sudo find /var/lib/k0s -mindepth 1 -delete; fi sudo rm -rf /etc/k0sOn a host where k0s is not running,
k0s resetcan take 20 minutes or longer to complete. Do not interrupt it. To keep a dropped SSH session from interrupting it, run it in a terminal multiplexer, such astmuxorscreen. -
Reboot the host to complete the reset.
-
-
If scheduled defragmentation is enabled for the etcd maintenance service, disable it. For more information, refer to Scheduled etcd defragmentation.
-
Prepare a hosts file that lists the new controllers with the
controller+workerrole. Do not include the surviving controller, which serves as the helper controller for the operation.hosts: - role: controller+worker ssh: address: <controller-2-ip-address> user: <SSH-user-name> port: <ssh port> keyPath: <full-path-to-SSH-private-key> - role: controller+worker ssh: address: <controller-3-ip-address> user: <SSH-user-name> port: <ssh port> keyPath: <full-path-to-SSH-private-key> -
Add the new controllers in a single run, so that the cluster returns to an odd number of controllers:
mkectl node add --scope="controller+worker" --hosts-path="<hosts-file>" \ --ssh-key="<path to ssh key to be used to SSH to survivor controller>" \ --timeout-minutes=0 \ --readiness-timeout-minutes=0 \ --passphrase - # read backup passphrase from stdinmkectlreaches the Kubernetes API through the external address of the cluster, which must route to the surviving controller. It joins the controllers one at a time and takes a cluster backup before and after the operation. To leave the backups unencrypted, pass--no-passphraseinstead of--passphrase -. For more information, refer to Add nodes and Backups.Note
mkectl node addfails its pre-flight check withfirewalld is active or enabled on: ...if firewalld runs on any of the new hosts, even when firewalld is configured for MKE 4.k0s resetdoes not remove the firewalld configuration of a reused host. If firewalld is configured for MKE 4 on each new host, add the--skip-firewalld-checkflag to the command. -
If the cluster uses an external load balancer, verify that its backend list includes the surviving controller and every replacement controller, and remove the backends of controllers that are no longer in the cluster. If the replacement controllers reuse the addresses of the lost controllers, the list is already correct. Verify that the Kubernetes API is reachable through the external address of the cluster.
-
If you disabled scheduled defragmentation in step 2, re-enable it.
Verify the recovered cluster#
-
Verify that every controller is joined to etcd:
sudo curl --cacert /var/lib/k0s/pki/etcd/ca.crt --cert /var/lib/k0s/pki/etcd/server.crt --key /var/lib/k0s/pki/etcd/server.key -s https://127.0.0.1:2379/v3/cluster/member/list -X POST | jqExample output:
{ "header": { "cluster_id": "7346508586475956732", "member_id": "11132754253862912738", "raft_term": "183" }, "members": [ { "ID": "1139358203818174176", "name": "controller-3", "peerURLs": [ "https://100.64.100.7:2380" ], "clientURLs": [ "https://127.0.0.1:2379" ] }, { "ID": "8600241746807059493", "name": "controller-2", "peerURLs": [ "https://100.64.100.6:2380" ], "clientURLs": [ "https://127.0.0.1:2379" ] }, { "ID": "11132754253862912738", "name": "controller-1", "peerURLs": [ "https://100.64.100.5:2380" ], "clientURLs": [ "https://127.0.0.1:2379" ] } ] } -
Verify that every controller has an
EtcdMemberobject withJOINEDset toTrue, and that no object remains for a lost controller:kubectl get etcdmembersExample output:
NAME PEER ADDRESS MEMBER ID JOINED RECONCILE STATUS controller-1 100.64.100.5 9a7f7d029137c2e2 True controller-2 100.64.100.6 775a324d7ceb7425 True controller-3 100.64.100.7 fcfd03ff28266e0 True -
Verify that every node is
Readyand that none of the lost nodes is listed:kubectl get nodesExample output:
NAME STATUS ROLES AGE VERSION controller-1 Ready control-plane 8d v1.35.4+k0s controller-2 Ready control-plane 7m16s v1.35.4+k0s controller-3 Ready control-plane 7m2s v1.35.4+k0s worker-1 Ready <none> 8d v1.35.4+k0s -
Verify that the MKE 4 system pods are running:
kubectl get pods -n mke -
Create a new backup of the recovered cluster. If you use scheduled backups, verify that they run.