Node scenarios#
Before acting on a host, mkectl inspects its current state to determine
whether k0s is installed and running, whether the node is part of this or
another cluster, and whether the node is registered and healthy in the
Kubernetes API. Based on this, a determination is made as to whether to proceed
with the operation, skip the host, or fail with an error.
The table below covers all supported scenarios. The "CA fingerprint match" column group from the design is split into "Part of this cluster" and "Part of any cluster". Membership is detected by comparing the cluster CA fingerprint found on the host against the one from the current cluster.
Note that control plane add/remove scenarios also split by etcd member verdict(s) (refer to etcd member verdicts)
Node Lifecycle Scenario Matrix
| CA fingerprint match | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| # | Scenario | k0s installed | k0s healthy | Part of this cluster | Part of any cluster | Node registered in cluster | Node ready | ETCD cluster healthy and out-of-scope members are healthy | Worker node add action | Worker node remove action | Etcd verdict (most strict verdict used across scoped hosts) |
CP node add action | CP node remove action | |
| 0 | Clean host with k0s binary installed | Yes | No (not running, no service installed) | No (error "no traces of cluster CA" is ignored) | No (error "no traces of cluster CA" is ignored) | No (not checked) | – | Yes | happy path | k0s reset (best effort) | a. VerdictClean / missed from list | happy path | k0s reset (best effort) | |
| b. VerdictMemberLeftCRNeedsCleanup | cleanup stale CR, then happy path | cleanup stale CR, k0s reset (best effort) | ||||||||||||
| c. VerdictLeftCRNotUpdated | report, fail | exclude from etcd removal; wait for CR to reach Joined=False (best effort?); remove CR? | ||||||||||||
| d. VerdictHealthy | report, fail | happy path, skip node drain/cordon, k0s reset best effort, cleanup | ||||||||||||
| e. VerdictLeaveRequested | report, fail | wait until member left etcd, skip node cordon and drain, k0s reset best effort, cleanup | ||||||||||||
| f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown | report, fail | report, fail | ||||||||||||
| 1 | Clean host | No | – | No (check skipped since k0s is not installed) | No (check skipped since k0s is not installed) | No (not checked) | – | Yes | happy path | report; de-register from MKEConfig only | a. VerdictClean / missed from list | happy path | k0s reset (best effort) | |
| b. VerdictMemberLeftCRNeedsCleanup | cleanup stale CR, then happy path | cleanup stale CR, k0s reset (best effort) | ||||||||||||
| c. VerdictLeftCRNotUpdated | report, fail | exclude from etcd removal; wait for CR to reach Joined=False (best effort?); remove CR? | ||||||||||||
| d. VerdictHealthy | report, fail | happy path, skip node drain/cordon, k0s reset best effort, cleanup | ||||||||||||
| e. VerdictLeaveRequested | report, fail | wait until member left etcd, skip node cordon and drain, k0s reset best effort, cleanup | ||||||||||||
| f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown | report, fail | report, fail | ||||||||||||
| 2 | k0s installed, part of this cluster, node ready | Yes | Yes | Yes | Yes | Yes | Yes | Yes | report; register in MKEConfig only | happy path | a. VerdictClean / missed from list | report, fail | skip etcd removal, happy path | |
| b. VerdictHealthy | report; register in MKEConfig only | happy path | ||||||||||||
| c. VerdictLeaveRequested | report, fail | skip etcd removal, wait for member to leave etcd, happy path | ||||||||||||
| d. VerdictLeftCRNotUpdated | report, fail | skip etcd removal, wait for CR to reach Joined=False, remove CR, happy path | ||||||||||||
| e. VerdictMemberLeftCRNeedsCleanup | report, fail | skip etcd removal, cleanup CR, happy path | ||||||||||||
| f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown | report, fail | report, fail | ||||||||||||
| 3 | k0s installed, part of this cluster, node NOT ready | Yes | Yes | Yes | Yes | Yes | No | Yes | exclude from adding, but wait for readiness | happy path | a. VerdictClean / missed from list | report, fail | skip removal from etcd, happy path | |
| b. VerdictHealthy | exclude from adding, but wait for readiness | happy path | ||||||||||||
| c. VerdictLeaveRequested | report, fail | skip etcd removal, wait for member to leave etcd, happy path | ||||||||||||
| d. VerdictLeftCRNotUpdated | report, fail | skip etcd removal, wait for CR to reach Joined=False, remove CR, happy path | ||||||||||||
| e. VerdictMemberLeftCRNeedsCleanup | report, fail | skip etcd removal, cleanup CR, happy path | ||||||||||||
| f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown | report, fail | report, fail | ||||||||||||
| 4 | k0s installed, part of this cluster, node NOT registered | Yes | Yes | Yes | Yes | No | – | Yes | restart k0sworker service, wait for node readiness | report, k0s reset (best effort) | a. VerdictClean / missed from list | report, fail | skip removal from etcd, happy path (exclude cordon, drain) | |
| b. VerdictHealthy | restart k0scontroller service, wait for node readiness | happy path (exclude cordon, drain) | ||||||||||||
| c. VerdictLeaveRequested | report, fail | skip etcd removal, wait for member to leave etcd, happy path (exclude cordon, drain) | ||||||||||||
| d. VerdictLeftCRNotUpdated | report, fail | skip etcd removal, wait for CR to reach Joined=False, remove CR, happy path (exclude cordon, drain) | ||||||||||||
| e. VerdictMemberLeftCRNeedsCleanup | report, fail | skip etcd removal, cleanup CR, happy path (exclude cordon, drain) | ||||||||||||
| f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown | report, fail | report, fail | ||||||||||||
| 5 | k0s installed, NOT part of this cluster, but part of another cluster | Yes | Yes | No | Yes | Doesn't matter | Doesn't matter | Yes | report, fail | report, fail | Etcd verdict detection shouldn't run; early fail | report, fail | report, fail | |
| 6 | k0s installation corrupted, part of this cluster, node registered in cluster | Yes | No | Yes | Yes | Yes | No (can't be ready) | Yes | report, fail | k0s reset (best effort) + node removal | a. VerdictClean / missed from list | report, fail | skip etcd leave, k0s reset (best effort) + node removal | |
| b. VerdictHealthy | report, fail | leave etcd, wait for leaving, cleanup CR, node cordon and drain, k0s reset best effort | ||||||||||||
| c. VerdictLeaveRequested | report, fail | wait for etcd member to leave etcd (best effort); if waiting failed, fail — otherwise proceed to node cordon, drain, and k0s reset best effort | ||||||||||||
| d. VerdictLeftCRNotUpdated | report, fail | wait for CR to have Joined=False (best effort); if waiting failed, fail — otherwise skip leaving etcd and proceed to node cordon, drain, and k0s reset best effort | ||||||||||||
| e. VerdictMemberLeftCRNeedsCleanup | report, fail | skip etcd leave and waiting, remove CR, proceed to node cordon, drain, and k0s reset best effort | ||||||||||||
| f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown | report, fail | report, fail | ||||||||||||
| 7 | k0s installation corrupted, part of this cluster, node NOT registered in cluster | Yes | No | Yes | Yes | No | – | Yes | report, fail | k0s reset (best effort) | a. VerdictClean / missed from list | report, fail | skip etcd leave, k0s reset (best effort) | |
| b. VerdictHealthy | report, fail | leave etcd, wait for leaving, cleanup CR, skip node cordon and drain, k0s reset best effort | ||||||||||||
| c. VerdictLeaveRequested | report, fail | wait for etcd member to leave etcd (best effort); if waiting failed, fail — otherwise proceed to k0s reset best effort | ||||||||||||
| d. VerdictLeftCRNotUpdated | report, fail | wait for CR to have Joined=False (best effort); if waiting failed, fail — otherwise skip leaving etcd and proceed to k0s reset best effort | ||||||||||||
| e. VerdictMemberLeftCRNeedsCleanup | report, fail | skip etcd leave and waiting, remove CR, proceed to k0s reset best effort | ||||||||||||
| f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown | report, fail | report, fail | ||||||||||||
| 8 | k0s installation corrupted, NOT part of this cluster | Yes | No | No | Yes | Doesn't matter | Doesn't matter | Doesn't matter, early fail | report, fail | report, fail | Etcd verdict detection shouldn't run; early fail | report, fail | report, fail | |
| 9 | k0s installation corrupted, NOT part of ANY cluster | Yes | No (either not running or no service installed) | Can't detect (error: no traces of cluster CA) | Can't detect (error: no traces of cluster CA) | Doesn't matter | Doesn't matter | Doesn't matter, early fail | report, fail | report, fail | Etcd verdict detection shouldn't run; early fail | report, fail | report, fail | |
| 10 | k0s not installed, part of this cluster | No | No | Yes | Yes | Doesn't matter | Doesn't matter | Yes | report, fail (needs manual cleanup) | cleanup folders | a. VerdictClean / missed from list | report, fail | cleanup folders | |
| b. VerdictHealthy | report, fail | leave etcd, wait for leaving, cleanup CR, skip node cordon and drain, cleanup folders | ||||||||||||
| c. VerdictLeaveRequested | report, fail | wait for etcd member to leave etcd (best effort); if waiting failed, fail — otherwise proceed to cleanup folders | ||||||||||||
| d. VerdictLeftCRNotUpdated | report, fail | wait for CR to have Joined=False (best effort); if waiting failed, fail — otherwise skip leaving etcd and proceed to cleanup folders | ||||||||||||
| e. VerdictMemberLeftCRNeedsCleanup | report, fail | skip etcd leave and waiting, remove CR, proceed to folder cleanup | ||||||||||||
| f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown | report, fail | report, fail | ||||||||||||