The rescheduling of stateful applications may be required when replacing a permanently failed node, decomissioning a node, migrating applications to nodes with a more suitable set of hardware, and in several other use cases.
MOS deployment profiles include the following stateful applications:
OpenStack database (MariaDB)
OpenStack coordination (etcd)
OpenStack Time Series Database back end (Redis)
Each stateful application from the list above has a persistent volume claim (PVC) based on a local persistent volume per pod. Each of control plane nodes has a set of local volumes available. To migrate an application pod to another node, recreate a PVC with the persistent volume from the target node.
Caution
A stateful application pod can only be migrated to a node that does not contain other pods of this application.
Caution
When a PVC is removed, all data present in the related persistent volume is removed from the node as well.
To reschedule pods to another control plane node:
Recreate a PVC on another control plane node:
Select one of the persistent volumes available on the node:
Caution
A stateful application pod can only be migrated to the node that does not contain other pods of this application.
NODE_NAME=<NODE-NAME>
STORAGE_CLASS=$(kubectl -n openstack get osdpl <OSDPL_OBJECT_NAME> -o jsonpath='{.spec.local_volume_storage_class}')
kubectl -n openstack get pv -o json | jq --arg NODE_NAME $NODE_NAME --arg STORAGE_CLASS $STORAGE_CLASS -r '.items[] | select(.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0] == $NODE_NAME and .spec.storageClassName == $STORAGE_CLASS and .status.phase == "Available") | .metadata.name'
As the new PVC should contain the same parameters as the deleted
one except for volumeName
, save the old PVC configuration in YAML:
kubectl -n <NAMESPACE> get pvc <PVC-NAME> -o yaml > <OLD-PVC>.yaml
Note
<NAMESPACE>
is a Kubernetes namespace where the PVC
is created. For Redis, specify openstack-redis
, for other
applications specify openstack
.
Delete the old PVC:
kubectl -n <NAMESPACE> delete pvc <PVC-NAME>
Note
If a PVC has stuck in the terminating
state, run
kubectl -n openstack edit pvc <PVC-NAME>
and remove the
finalizers
section from metadata
of the PVC.
Create the PVC with a new persistent volume:
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: <PVC-NAME>
namespace: <NAMESPACE>
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: <STORAGE-SIZE>
storageClassName: <STORAGE-CLASS>
volumeMode: Filesystem
volumeName: <PV-NAME>
EOF
Caution
<STORAGE-SIZE>
, <STORAGE-CLASS>
, and
<NAMESPACE>
should correspond to the storage
,
storageClassName
, and namespace
values from the
<OLD-PVC>.yaml
file with the old PVC configuration.
Reschedule the pods:
For MariaDB:
Remove the pod:
Note
To remove a pod from a node in the NotReady
state, add
--grace-period=0 --force
to the following command.
kubectl -n openstack delete pod <STATEFULSET-NAME>-<NUMBER>
Wait until the pod appears in the Ready
state.
When the rescheduling is finalized, the mariadb-server-2
pod
joins back to the Galera cluster with a clean MySQL data directory
and requests the Galera state transfer from the available nodes.
For etcd:
Scale down the etcd StatefulSet to N-1 replicas, where N is the initial number of replicas in the etcd StatefulSet:
kubectl -n openstack scale sts etcd-etcd --replicas=<N-1>
Add the coordination
section to the spec.services
section
of the OsDpl object:
spec:
services:
coordination:
etcd:
values:
conf:
etcd:
ETCD_INITIAL_CLUSTER_STATE: existing
Wait until all etcd pods appear in the Ready
state.
Scale the etcd StatefulSet to the initial number of replicas:
kubectl -n openstack scale sts etcd-etcd --replicas=<NUMBER-OF-REPLICAS>
Wait until all etcd pods appear in the Ready
state.
Remove the coordination
section from the spec.services
section of the OsDpl object.
Wait until all etcd pods appear in the Ready
state.
For Redis:
Remove the pod:
Note
To remove a pod from a node in the NotReady
state, add
--grace-period=0 --force
to the following command.
kubectl -n openstack-redis delete pod <STATEFULSET-NAME>-<NUMBER>
Wait until the pod appears in the Ready
state.