- Home
- Configuration
- Scale nodes
- Node scenarios
Node scenarios#
Before mkectl acts on a host, it inspects the current state of the host and
then proceeds with the operation, skips the host, or fails with an error. The
inspection determines:
- Whether k0s is installed and running on the host
- Whether the host belongs to this cluster, another cluster, or no cluster
- Whether the node is registered and healthy in the Kubernetes API
mkectl determines cluster membership by comparing the cluster CA fingerprint
on the host with the CA fingerprint of the current cluster. The
Part of this cluster and Part of any cluster columns in the scenario
table reflect the result of this comparison.
A host has k0s traces when any of the following conditions is true:
- k0s is running on the host.
- A k0s service is installed on the host.
- The k0s data directory contains anything other than empty directories.
A host with k0s traces but no cluster CA certificate falls under scenario 9, even if k0s is not installed. If the data directory is the only trace, the error message lists the directory contents. For more information, refer to k0s data directory.
Note
Scenarios that add or remove control plane nodes also depend on the etcd member verdict for the host. For more information, refer to etcd member verdicts.
The following table describes all supported scenarios:
Node Lifecycle Scenario Matrix
| CA fingerprint match | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| # | Scenario | k0s installed | k0s healthy | Part of this cluster | Part of any cluster | Node registered in cluster | Node ready | ETCD cluster healthy and out-of-scope members are healthy | Worker node add action | Worker node remove action | Etcd verdict (most strict verdict used across scoped hosts) |
CP node add action | CP node remove action | |
| 0 | Clean host with k0s binary installed | Yes | No (not running, no service installed) | No (error "no traces of cluster CA" is ignored) | No (error "no traces of cluster CA" is ignored) | No (not checked) | – | Yes | happy path | k0s reset (best effort) | a. VerdictClean / missed from list | happy path | k0s reset (best effort) | |
| b. VerdictMemberLeftCRNeedsCleanup | cleanup stale CR, then happy path | cleanup stale CR, k0s reset (best effort) | ||||||||||||
| c. VerdictLeftCRNotUpdated | report, fail | exclude from etcd removal; wait for CR to reach Joined=False (best effort?); remove CR? | ||||||||||||
| d. VerdictHealthy | report, fail | happy path, skip node drain/cordon, k0s reset best effort, cleanup | ||||||||||||
| e. VerdictLeaveRequested | report, fail | wait until member left etcd, skip node cordon and drain, k0s reset best effort, cleanup | ||||||||||||
| f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown | report, fail | report, fail | ||||||||||||
| 1 | Clean host | No | – | No (check skipped since k0s is not installed) | No (check skipped since k0s is not installed) | No (not checked) | – | Yes | happy path | report; de-register from MKEConfig only | a. VerdictClean / missed from list | happy path | k0s reset (best effort) | |
| b. VerdictMemberLeftCRNeedsCleanup | cleanup stale CR, then happy path | cleanup stale CR, k0s reset (best effort) | ||||||||||||
| c. VerdictLeftCRNotUpdated | report, fail | exclude from etcd removal; wait for CR to reach Joined=False (best effort?); remove CR? | ||||||||||||
| d. VerdictHealthy | report, fail | happy path, skip node drain/cordon, k0s reset best effort, cleanup | ||||||||||||
| e. VerdictLeaveRequested | report, fail | wait until member left etcd, skip node cordon and drain, k0s reset best effort, cleanup | ||||||||||||
| f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown | report, fail | report, fail | ||||||||||||
| 2 | k0s installed, part of this cluster, node ready | Yes | Yes | Yes | Yes | Yes | Yes | Yes | report; register in MKEConfig only | happy path | a. VerdictClean / missed from list | report, fail | skip etcd removal, happy path | |
| b. VerdictHealthy | report; register in MKEConfig only | happy path | ||||||||||||
| c. VerdictLeaveRequested | report, fail | skip etcd removal, wait for member to leave etcd, happy path | ||||||||||||
| d. VerdictLeftCRNotUpdated | report, fail | skip etcd removal, wait for CR to reach Joined=False, remove CR, happy path | ||||||||||||
| e. VerdictMemberLeftCRNeedsCleanup | report, fail | skip etcd removal, cleanup CR, happy path | ||||||||||||
| f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown | report, fail | report, fail | ||||||||||||
| 3 | k0s installed, part of this cluster, node NOT ready | Yes | Yes | Yes | Yes | Yes | No | Yes | exclude from adding, but wait for readiness | happy path | a. VerdictClean / missed from list | report, fail | skip removal from etcd, happy path | |
| b. VerdictHealthy | exclude from adding, but wait for readiness | happy path | ||||||||||||
| c. VerdictLeaveRequested | report, fail | skip etcd removal, wait for member to leave etcd, happy path | ||||||||||||
| d. VerdictLeftCRNotUpdated | report, fail | skip etcd removal, wait for CR to reach Joined=False, remove CR, happy path | ||||||||||||
| e. VerdictMemberLeftCRNeedsCleanup | report, fail | skip etcd removal, cleanup CR, happy path | ||||||||||||
| f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown | report, fail | report, fail | ||||||||||||
| 4 | k0s installed, part of this cluster, node NOT registered | Yes | Yes | Yes | Yes | No | – | Yes | restart k0sworker service, wait for node readiness | report, k0s reset (best effort) | a. VerdictClean / missed from list | report, fail | skip removal from etcd, happy path (exclude cordon, drain) | |
| b. VerdictHealthy | restart k0scontroller service, wait for node readiness | happy path (exclude cordon, drain) | ||||||||||||
| c. VerdictLeaveRequested | report, fail | skip etcd removal, wait for member to leave etcd, happy path (exclude cordon, drain) | ||||||||||||
| d. VerdictLeftCRNotUpdated | report, fail | skip etcd removal, wait for CR to reach Joined=False, remove CR, happy path (exclude cordon, drain) | ||||||||||||
| e. VerdictMemberLeftCRNeedsCleanup | report, fail | skip etcd removal, cleanup CR, happy path (exclude cordon, drain) | ||||||||||||
| f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown | report, fail | report, fail | ||||||||||||
| 5 | k0s installed, NOT part of this cluster, but part of another cluster | Yes | Yes | No | Yes | Doesn't matter | Doesn't matter | Yes | report, fail | report, fail | Etcd verdict detection shouldn't run; early fail | report, fail | report, fail | |
| 6 | k0s installation corrupted, part of this cluster, node registered in cluster | Yes | No | Yes | Yes | Yes | No (can't be ready) | Yes | report, fail | k0s reset (best effort) + node removal | a. VerdictClean / missed from list | report, fail | skip etcd leave, k0s reset (best effort) + node removal | |
| b. VerdictHealthy | report, fail | leave etcd, wait for leaving, cleanup CR, node cordon and drain, k0s reset best effort | ||||||||||||
| c. VerdictLeaveRequested | report, fail | wait for etcd member to leave etcd (best effort); if waiting failed, fail — otherwise proceed to node cordon, drain, and k0s reset best effort | ||||||||||||
| d. VerdictLeftCRNotUpdated | report, fail | wait for CR to have Joined=False (best effort); if waiting failed, fail — otherwise skip leaving etcd and proceed to node cordon, drain, and k0s reset best effort | ||||||||||||
| e. VerdictMemberLeftCRNeedsCleanup | report, fail | skip etcd leave and waiting, remove CR, proceed to node cordon, drain, and k0s reset best effort | ||||||||||||
| f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown | report, fail | report, fail | ||||||||||||
| 7 | k0s installation corrupted, part of this cluster, node NOT registered in cluster | Yes | No | Yes | Yes | No | – | Yes | report, fail | k0s reset (best effort) | a. VerdictClean / missed from list | report, fail | skip etcd leave, k0s reset (best effort) | |
| b. VerdictHealthy | report, fail | leave etcd, wait for leaving, cleanup CR, skip node cordon and drain, k0s reset best effort | ||||||||||||
| c. VerdictLeaveRequested | report, fail | wait for etcd member to leave etcd (best effort); if waiting failed, fail — otherwise proceed to k0s reset best effort | ||||||||||||
| d. VerdictLeftCRNotUpdated | report, fail | wait for CR to have Joined=False (best effort); if waiting failed, fail — otherwise skip leaving etcd and proceed to k0s reset best effort | ||||||||||||
| e. VerdictMemberLeftCRNeedsCleanup | report, fail | skip etcd leave and waiting, remove CR, proceed to k0s reset best effort | ||||||||||||
| f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown | report, fail | report, fail | ||||||||||||
| 8 | k0s installation corrupted, NOT part of this cluster | Yes | No | No | Yes | Doesn't matter | Doesn't matter | Doesn't matter, early fail | report, fail | report, fail | Etcd verdict detection shouldn't run; early fail | report, fail | report, fail | |
| 9 | k0s installation corrupted, NOT part of ANY cluster | Yes | No (either not running or no service installed) | Can't detect (error: no traces of cluster CA) | Can't detect (error: no traces of cluster CA) | Doesn't matter | Doesn't matter | Doesn't matter, early fail | report, fail | report, fail | Etcd verdict detection shouldn't run; early fail | report, fail | report, fail | |
| 10 | k0s not installed, part of this cluster | No | No | Yes | Yes | Doesn't matter | Doesn't matter | Yes | report, fail (needs manual cleanup) | cleanup folders | a. VerdictClean / missed from list | report, fail | cleanup folders | |
| b. VerdictHealthy | report, fail | leave etcd, wait for leaving, cleanup CR, skip node cordon and drain, cleanup folders | ||||||||||||
| c. VerdictLeaveRequested | report, fail | wait for etcd member to leave etcd (best effort); if waiting failed, fail — otherwise proceed to cleanup folders | ||||||||||||
| d. VerdictLeftCRNotUpdated | report, fail | wait for CR to have Joined=False (best effort); if waiting failed, fail — otherwise skip leaving etcd and proceed to cleanup folders | ||||||||||||
| e. VerdictMemberLeftCRNeedsCleanup | report, fail | skip etcd leave and waiting, remove CR, proceed to folder cleanup | ||||||||||||
| f. VerdictHazardCRMissing, VerdictOOBRemovalStaleCR, VerdictCorruptedRemoval, VerdictCRStatusNeverWritten, VerdictCRStatusStale, VerdictUnknown | report, fail | report, fail | ||||||||||||