Deschedule StackLight Pods from a worker machine

On an existing managed cluster, addition of a worker machine that replaces the one containing the StackLight node label requires the label migration to the new machine and a manual removal of StackLight Pods from the old machine, which you remove the label from.

Caution

In this procedure, replace <machine-name> with the name of the machine from which you remove the StackLight node label.

To deschedule StackLight Pods from a worker machine:

  1. Remove the stacklight=enabled node label from the target Machine object.

  2. Connect to the required cluster using its kubeconfig.

  3. Verify that the stacklight=enabled label was removed successfully:

    kubectl get node -l "kaas.mirantis.com/machine-name=<machine name>" --show-labels | grep "stacklight=enabled"

    A positive system response must be empty.

  4. Verify the list of StackLight Pods to be deleted that run on the target machine:

    kubectl get pods -n stacklight -o wide --field-selector spec.nodeName=$(kubectl get node -l "kaas.mirantis.com/machine-name=<machine name>" -o jsonpath='{.items[0].metadata.name}')

    Example of system response extract:

    NAME                                           READY STATUS    AGE   IP             NODE
alerta-fc45c8f6-6qlfx                          1/1   Running   63m   10.233.76.3    node-3a0de232-c1b4-43b0-8f21-44cd1
grafana-9bc56cdff-sl5w6                        3/3   Running   63m   10.233.76.4    node-3a0de232-c1b4-43b0-8f21-44cd1
iam-proxy-alerta-57585798d7-kqwd7              1/1   Running   58m   10.233.76.17   node-3a0de232-c1b4-43b0-8f21-44cd1
iam-proxy-alertmanager-6b4c4c8867-pdwcs        1/1   Running   56m   10.233.76.18   node-3a0de232-c1b4-43b0-8f21-44cd1
iam-proxy-grafana-87b984c45-2qwvb              1/1   Running   55m   10.233.76.19   node-3a0de232-c1b4-43b0-8f21-44cd1
iam-proxy-prometheus-545789585-9mll8           1/1   Running   54m   10.233.76.21   node-3a0de232-c1b4-43b0-8f21-44cd1
patroni-13-0                                   3/3   Running   61m   10.233.76.11   node-3a0de232-c1b4-43b0-8f21-44cd1
prometheus-alertmanager-0                      1/1   Running   55m   10.233.76.20   node-3a0de232-c1b4-43b0-8f21-44cd1
prometheus-blackbox-exporter-9f6bdfd75-8zn4w   2/2   Running   61m   10.233.76.8    node-3a0de232-c1b4-43b0-8f21-44cd1
prometheus-kube-state-metrics-67ff88649f-tslxc 1/1   Running   61m   10.233.76.7    node-3a0de232-c1b4-43b0-8f21-44cd1
prometheus-node-exporter-zl8pj                 1/1   Running   61m   10.10.10.143   node-3a0de232-c1b4-43b0-8f21-44cd1
telegraf-docker-swarm-69567fcf7f-jvbgn         1/1   Running   61m   10.233.76.10   node-3a0de232-c1b4-43b0-8f21-44cd1
telemeter-client-55d465dcc5-9thds              1/1   Running   61m   10.233.76.9    node-3a0de232-c1b4-43b0-8f21-44cd1

  5. Delete all StackLight Pods from the target machine:

    kubectl -n stacklight delete $(kubectl get pods -n stacklight -o wide --field-selector spec.nodeName=$(kubectl get node -l "kaas.mirantis.com/machine-name=<machine name>" -o jsonpath='{.items[0].metadata.name}') -o name)

    Example of system response:

    pod "alerta-fc45c8f6-6qlfx" deleted
pod "grafana-9bc56cdff-sl5w6" deleted
pod "iam-proxy-alerta-57585798d7-kqwd7" deleted
pod "iam-proxy-alertmanager-6b4c4c8867-pdwcs" deleted
pod "iam-proxy-grafana-87b984c45-2qwvb" deleted
pod "iam-proxy-prometheus-545789585-9mll8" deleted
pod "patroni-13-0" deleted
pod "prometheus-alertmanager-0" deleted
pod "prometheus-blackbox-exporter-9f6bdfd75-8zn4w" deleted
pod "prometheus-kube-state-metrics-67ff88649f-tslxc" deleted
pod "prometheus-node-exporter-zl8pj" deleted
pod "telegraf-docker-swarm-69567fcf7f-jvbgn" deleted
pod "telemeter-client-55d465dcc5-9thds" deleted

  6. Wait about three minutes for Pods to be rescheduled.

  7. Verify that you do not have Pending Pods in the stacklight namespace:

    kubectl -n stacklight get pods --field-selector status.phase=Pending

    • If the system response is No resources found in stacklight namespace, all Pods are rescheduled successfully.

    • If the system response still contains some Pods, remove local persistent volumes bound to the target machine as described in Delete a cluster machine using web UI.