Skip to content

Node scenarios#

Before mkectl acts on a host, it inspects the current state of the host and then proceeds with the operation, skips the host, or fails with an error. The inspection determines:

  • Whether k0s is installed and running on the host
  • Whether the host belongs to this cluster, another cluster, or no cluster
  • Whether the node is registered and healthy in the Kubernetes API

mkectl determines cluster membership by comparing the cluster CA fingerprint on the host with the CA fingerprint of the current cluster. The Part of this cluster and Part of any cluster columns in the scenario table reflect the result of this comparison.

A host has k0s traces when any of the following conditions is true:

  • k0s is running on the host.
  • A k0s service is installed on the host.
  • The k0s data directory contains anything other than empty directories.

A host with k0s traces but no cluster CA certificate falls under scenario 9, even if k0s is not installed. If the data directory is the only trace, the error message lists the directory contents. For more information, refer to k0s data directory.

Note

Scenarios that add or remove control plane nodes also depend on the etcd member verdict for the host. For more information, refer to etcd member verdicts.

The following table describes all supported scenarios:

Node Lifecycle Scenario Matrix

Node Lifecycle Scenario Matrix
CA fingerprint match
# Scenario k0s installed k0s healthy Part of this cluster Part of any cluster Node registered in cluster Node ready ETCD cluster healthy and out-of-scope members are healthy Worker node add action Worker node remove action Etcd verdict
(most strict verdict used across scoped hosts)
CP node add action CP node remove action
0 Clean host with k0s binary installed Yes No (not running, no service installed) No (error "no traces of cluster CA" is ignored) No (error "no traces of cluster CA" is ignored) No (not checked) – Yes happy path k0s reset (best effort) a. VerdictClean / missed from list happy path k0s reset (best effort)
b. VerdictMemberLeftCRNeedsCleanup cleanup stale CR, then happy path cleanup stale CR, k0s reset (best effort)
c. VerdictLeftCRNotUpdated report, fail exclude from etcd removal; wait for CR to reach Joined=False (best effort?); remove CR?
d. VerdictHealthy report, fail happy path, skip node drain/cordon, k0s reset best effort, cleanup
e. VerdictLeaveRequested report, fail wait until member left etcd, skip node cordon and drain, k0s reset best effort, cleanup
f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown report, fail report, fail
1 Clean host No – No (check skipped since k0s is not installed) No (check skipped since k0s is not installed) No (not checked) – Yes happy path report; de-register from MKEConfig only a. VerdictClean / missed from list happy path k0s reset (best effort)
b. VerdictMemberLeftCRNeedsCleanup cleanup stale CR, then happy path cleanup stale CR, k0s reset (best effort)
c. VerdictLeftCRNotUpdated report, fail exclude from etcd removal; wait for CR to reach Joined=False (best effort?); remove CR?
d. VerdictHealthy report, fail happy path, skip node drain/cordon, k0s reset best effort, cleanup
e. VerdictLeaveRequested report, fail wait until member left etcd, skip node cordon and drain, k0s reset best effort, cleanup
f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown report, fail report, fail
2 k0s installed, part of this cluster, node ready Yes Yes Yes Yes Yes Yes Yes report; register in MKEConfig only happy path a. VerdictClean / missed from list report, fail skip etcd removal, happy path
b. VerdictHealthy report; register in MKEConfig only happy path
c. VerdictLeaveRequested report, fail skip etcd removal, wait for member to leave etcd, happy path
d. VerdictLeftCRNotUpdated report, fail skip etcd removal, wait for CR to reach Joined=False, remove CR, happy path
e. VerdictMemberLeftCRNeedsCleanup report, fail skip etcd removal, cleanup CR, happy path
f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown report, fail report, fail
3 k0s installed, part of this cluster, node NOT ready Yes Yes Yes Yes Yes No Yes exclude from adding, but wait for readiness happy path a. VerdictClean / missed from list report, fail skip removal from etcd, happy path
b. VerdictHealthy exclude from adding, but wait for readiness happy path
c. VerdictLeaveRequested report, fail skip etcd removal, wait for member to leave etcd, happy path
d. VerdictLeftCRNotUpdated report, fail skip etcd removal, wait for CR to reach Joined=False, remove CR, happy path
e. VerdictMemberLeftCRNeedsCleanup report, fail skip etcd removal, cleanup CR, happy path
f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown report, fail report, fail
4 k0s installed, part of this cluster, node NOT registered Yes Yes Yes Yes No – Yes restart k0sworker service, wait for node readiness report, k0s reset (best effort) a. VerdictClean / missed from list report, fail skip removal from etcd, happy path (exclude cordon, drain)
b. VerdictHealthy restart k0scontroller service, wait for node readiness happy path (exclude cordon, drain)
c. VerdictLeaveRequested report, fail skip etcd removal, wait for member to leave etcd, happy path (exclude cordon, drain)
d. VerdictLeftCRNotUpdated report, fail skip etcd removal, wait for CR to reach Joined=False, remove CR, happy path (exclude cordon, drain)
e. VerdictMemberLeftCRNeedsCleanup report, fail skip etcd removal, cleanup CR, happy path (exclude cordon, drain)
f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown report, fail report, fail
5 k0s installed, NOT part of this cluster, but part of another cluster Yes Yes No Yes Doesn't matter Doesn't matter Yes report, fail report, fail Etcd verdict detection shouldn't run; early fail report, fail report, fail
6 k0s installation corrupted, part of this cluster, node registered in cluster Yes No Yes Yes Yes No (can't be ready) Yes report, fail k0s reset (best effort) + node removal a. VerdictClean / missed from list report, fail skip etcd leave, k0s reset (best effort) + node removal
b. VerdictHealthy report, fail leave etcd, wait for leaving, cleanup CR, node cordon and drain, k0s reset best effort
c. VerdictLeaveRequested report, fail wait for etcd member to leave etcd (best effort); if waiting failed, fail — otherwise proceed to node cordon, drain, and k0s reset best effort
d. VerdictLeftCRNotUpdated report, fail wait for CR to have Joined=False (best effort); if waiting failed, fail — otherwise skip leaving etcd and proceed to node cordon, drain, and k0s reset best effort
e. VerdictMemberLeftCRNeedsCleanup report, fail skip etcd leave and waiting, remove CR, proceed to node cordon, drain, and k0s reset best effort
f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown report, fail report, fail
7 k0s installation corrupted, part of this cluster, node NOT registered in cluster Yes No Yes Yes No – Yes report, fail k0s reset (best effort) a. VerdictClean / missed from list report, fail skip etcd leave, k0s reset (best effort)
b. VerdictHealthy report, fail leave etcd, wait for leaving, cleanup CR, skip node cordon and drain, k0s reset best effort
c. VerdictLeaveRequested report, fail wait for etcd member to leave etcd (best effort); if waiting failed, fail — otherwise proceed to k0s reset best effort
d. VerdictLeftCRNotUpdated report, fail wait for CR to have Joined=False (best effort); if waiting failed, fail — otherwise skip leaving etcd and proceed to k0s reset best effort
e. VerdictMemberLeftCRNeedsCleanup report, fail skip etcd leave and waiting, remove CR, proceed to k0s reset best effort
f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown report, fail report, fail
8 k0s installation corrupted, NOT part of this cluster Yes No No Yes Doesn't matter Doesn't matter Doesn't matter, early fail report, fail report, fail Etcd verdict detection shouldn't run; early fail report, fail report, fail
9 k0s installation corrupted, NOT part of ANY cluster Yes No (either not running or no service installed) Can't detect (error: no traces of cluster CA) Can't detect (error: no traces of cluster CA) Doesn't matter Doesn't matter Doesn't matter, early fail report, fail report, fail Etcd verdict detection shouldn't run; early fail report, fail report, fail
10 k0s not installed, part of this cluster No No Yes Yes Doesn't matter Doesn't matter Yes report, fail (needs manual cleanup) cleanup folders a. VerdictClean / missed from list report, fail cleanup folders
b. VerdictHealthy report, fail leave etcd, wait for leaving, cleanup CR, skip node cordon and drain, cleanup folders
c. VerdictLeaveRequested report, fail wait for etcd member to leave etcd (best effort); if waiting failed, fail — otherwise proceed to cleanup folders
d. VerdictLeftCRNotUpdated report, fail wait for CR to have Joined=False (best effort); if waiting failed, fail — otherwise skip leaving etcd and proceed to cleanup folders
e. VerdictMemberLeftCRNeedsCleanup report, fail skip etcd leave and waiting, remove CR, proceed to folder cleanup
f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown report, fail report, fail