Reschedule stateful applications¶

The rescheduling of stateful applications may be required when replacing a permanently failed node, decomissioning a node, migrating applications to nodes with a more suitable set of hardware, and in several other use cases.

MOS deployment profiles include the following stateful applications:

• OpenStack database (MariaDB)

• OpenStack coordination (etcd)

• OpenStack Time Series Database back end (Redis)

Each stateful application from the list above has a persistent volume claim (PVC) based on a local persistent volume per pod. Each of control plane nodes has a set of local volumes available. To migrate an application pod to another node, recreate a PVC with the persistent volume from the target node.

To reschedule pods to another control plane node:

1. Recreate a PVC on another control plane node:

   1. Select one of the persistent volumes available on the node:

      Caution

      A stateful application pod can only be migrated to the node that does not contain other pods of this application.

      NODE_NAME=<NODE-NAME>
      STORAGE_CLASS=$(kubectl -n openstack get osdpl <OSDPL_OBJECT_NAME> -o jsonpath='{.spec.local_volume_storage_class}')
      kubectl -n openstack get pv -o json | jq --arg NODE_NAME $NODE_NAME --arg STORAGE_CLASS $STORAGE_CLASS -r '.items[] | select(.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0] == $NODE_NAME and .spec.storageClassName == $STORAGE_CLASS and .status.phase == "Available") | .metadata.name'
      

   2. As the new PVC should contain the same parameters as the deleted one except for volumeName, save the old PVC configuration in YAML:

      kubectl -n <NAMESPACE> get pvc <PVC-NAME> -o yaml > <OLD-PVC>.yaml
      

      Note

      <NAMESPACE> is a Kubernetes namespace where the PVC is created. For Redis, specify openstack-redis, for other applications specify openstack.

   3. Delete the old PVC:

      kubectl -n <NAMESPACE> delete pvc <PVC-NAME>
      

      Note

      If a PVC has stuck in the terminating state, run kubectl -n openstack edit pvc <PVC-NAME> and remove the finalizers section from metadata of the PVC.

   4. Create the PVC with a new persistent volume:

      cat <<EOF | kubectl apply -f -
           apiVersion: v1
           kind: PersistentVolumeClaim
           metadata:
             name: <PVC-NAME>
             namespace: <NAMESPACE>
           spec:
             accessModes:
             - ReadWriteOnce
             resources:
               requests:
                 storage: <STORAGE-SIZE>
             storageClassName: <STORAGE-CLASS>
             volumeMode: Filesystem
             volumeName: <PV-NAME>
          EOF
      

      Caution

      <STORAGE-SIZE>, <STORAGE-CLASS>, and <NAMESPACE> should correspond to the storage, storageClassName, and namespace values from the <OLD-PVC>.yaml file with the old PVC configuration.

2. Reschedule the pods:

   • For MariaDB:

     1. Remove the pod:

        Note

        To remove a pod from a node in the NotReady state, add --grace-period=0 --force to the following command.

        kubectl -n openstack delete pod <STATEFULSET-NAME>-<NUMBER>
        

     2. Wait until the pod appears in the Ready state.

        When the rescheduling is finalized, the mariadb-server-2 pod joins back to the Galera cluster with a clean MySQL data directory and requests the Galera state transfer from the available nodes.

   • For etcd:

     1. Scale down the etcd StatefulSet to N-1 replicas, where N is the initial number of replicas in the etcd StatefulSet:

        kubectl -n openstack scale sts etcd-etcd --replicas=<N-1>
        

     2. Add the coordination section to the spec.services section of the OsDpl object:

        spec:
          services:
            coordination:
              etcd:
                values:
                  conf:
                    etcd:
                      ETCD_INITIAL_CLUSTER_STATE: existing
        

     3. Wait until all etcd pods appear in the Ready state.

     4. Scale the etcd StatefulSet to the initial number of replicas:

        kubectl -n openstack scale sts etcd-etcd --replicas=<NUMBER-OF-REPLICAS>
        

     5. Wait until all etcd pods appear in the Ready state.

     6. Remove the coordination section from the spec.services section of the OsDpl object.

     7. Wait until all etcd pods appear in the Ready state.

   • For Redis:

     1. Remove the pod:

        Note

        To remove a pod from a node in the NotReady state, add --grace-period=0 --force to the following command.

        kubectl -n openstack-redis delete pod <STATEFULSET-NAME>-<NUMBER>
        

     2. Wait until the pod appears in the Ready state.