Workflows of the OpenStack database backup and restoration¶
This section provides technical details about the internal implementation of automated backup and restoration routines built into MOSK. The below information would be helpful for troubleshooting of any issues related to the process or understanding the impact these procedures impose on a running cloud.
The OpenStack database backup workflow consists of the following phases.
Backup phase 1¶
mariadb-phy-backup job launches the
mariadb-phy-backup-<TIMESTAMP> pod. This pod contains the main backup
script, which is responsible for:
Basic sanity checks and choosing right node for backup
Verifying the wsrep status and changing the
During the first backup phase, the following actions take place:
Sanity check: verification of the Kubernetes status and wsrep status of each MariaDB pod. If some pods have wrong statuses, the backup job fails unless the
--allow-unsafe-backupparameter is passed to the main script in the Kubernetes backup job.
Since MOSK 22.4, the
--allow-unsafe-backupfunctionality is removed from the product for security and backup procedure simplification purposes.
Mirantis does not recommend setting the
--allow-unsafe-backupparameter unless it is absolutely required. To ensure the consistency of a backup, verify that the MariaDB Galera cluster is in a working state before you proceed with the backup.
Select the replica to back up. The system selects the replica with the highest number in its name as a target replica. For example, if the MariaDB server pods have the
mariadb-server-2replica will be backed up.
Desynchronize the replica from the Galera cluster. The script connects the target replica and sets the
ON. Then, the replica stops receiving write-sets and receives the wsrep status
Donor/Desynced. The Kubernetes health check of that
mariadb-serverpod fails and the Kubernetes status of that pod becomes
Not ready. If the pod has the
primarylabel, the MariaDB Controller sets the
backuplabel to it and the pod is removed from the endpoints list of the MariaDB service.
Backup phase 2¶
The main script in the
mariadb-phy-backuppod launches the Kubernetes pod
mariadb-phy-backup-runner-<TIMESTAMP>on the same node where the target
mariadb-serverreplica is running, which is node
Xin the example.
mariadb-phy-backup-runnerpod has both
mysqldata directory and
backupdirectory mounted. The pod performs the following actions:
Verifies that there is enough space in the
/var/backupfolder to perform the backup. The amount of available space in the folder should be greater than
<DB-SIZE> * <MARIADB-BACKUP-REQUIRED-SPACE-RATIOin KB.
Performs the actual backup using the mariabackup tool.
If the number of current backups is greater than the value of the
MARIADB_BACKUPS_TO_KEEPjob parameter, the script removes all old backups exceeding the allowed number of backups.
The script waits untill the
mariadb-phy-backup-runnerpod is completed and collects its logs.
The script puts the backed up replica back to sync with the Galera cluster by setting
OFFand waits for the replica to become
The OpenStack database restoration workflow consists of the following phases.
Restoration phase 1¶
mariadb-phy-restore job launches the
This pod contains the main restore script, which is responsible for:
Scaling of the
Verifying of the
Managing of the
During the restoration, the database is not available for OpenStack services that means a complete outage of all OpenStack services.
During the first phase, the following actions are performed:
Save the list of
mariadb-serverpersistent volume claims (PVC).
mariadbserver StatefulSet to
0replicas. At this point, the database becomes unavailable for OpenStack services.
Restoration phase 2¶
openstack-mariadb-phy-restore-runnerwith the first
mariadb-serverreplica PVC mounted to the
/var/lib/mysqlfolder and the backup PVC mounted to
openstack-mariadb-phy-restore-runnerpod performs the following actions:
Unarchives the database backup files to a temporary directory within
mariabackup --prepareon the unarchived data.
.preparedfile in the temporary directory in
Restores the backup to
Exits with 0.
The script in the
mariadb-phy-restorepod collects the logs from the
openstack-mariadb-phy-restore-runnerpod and removes the pod. Then, the script launches the next
openstack-mariadb-phy-restore-runnerpod for the next
mariadb-serverreplica PVC. The
openstack-mariadb-phy-restore-runnerpod restores the backup to
/var/lib/mysqland exits with
Step 2 is repeated for every
mariadb-serverreplica PVC sequentially.
When the last replica’s data is restored, the last
openstack-mariadb-phy-restore-runnerpod removes the
.preparedfile and the temporary folder with unachieved data from
Restoration phase 3¶
mariadb-phy-restorepod scales the
mariadb-serverStatefulSet back to the configured number of replicas.
mariadb-phy-restorepod waits until all
mariadb-serverreplicas are ready.