Restore OpenStack databases from a backup¶

During the OpenStack database restoration, the MariaDB cluster is unavailable due to the MariaDB StatefulSet being scaled down to 0 replicas. Therefore, to safely restore the state of the OpenStack database, plan the maintenance window thoroughly and in accordance with the database size.

The duration of the maintenance window may depend on the following:

Network throughput
Performance of the storage where backups are kept, which is Mirantis Ceph by default
Local disks performance of the nodes where MariaDB data resides

To restore OpenStack databases:

Obtain an image of the MariaDB container:

kubectl -n openstack get pods mariadb-server-0 -o jsonpath='{.spec.containers[0].image}'

Create the check_pod.yaml file to create the helper pod required to view the backup volume content:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: check-backup-helper
  namespace: openstack
---
apiVersion: v1
kind: Pod
metadata:
  name: check-backup-helper
  namespace: openstack
  labels:
    application: check-backup-helper
spec:
  nodeSelector:
    openstack-control-plane: enabled
  containers:
    - name: helper
      securityContext:
        allowPrivilegeEscalation: false
        runAsUser: 0
        readOnlyRootFilesystem: true
      command:
        - sleep
        - infinity
      image: << image of mariadb container >>
      imagePullPolicy: IfNotPresent
      volumeMounts:
        - name: pod-tmp
          mountPath: /tmp
        - mountPath: /var/backup
          name: mysql-backup
  restartPolicy: Never
  serviceAccount: check-backup-helper
  serviceAccountName: check-backup-helper
  volumes:
    - name: pod-tmp
      emptyDir: {}
    - name: mariadb-secrets
      secret:
        secretName: mariadb-secrets
        defaultMode: 0444
    - name: mariadb-bin
      configMap:
        name: mariadb-bin
        defaultMode: 0555
    - name: mysql-backup
      persistentVolumeClaim:
        claimName: mariadb-phy-backup-data

Create the helper pod:

kubectl -n openstack apply -f check_pod.yaml

Obtain the name of the backup to restore:
```
kubectl -n openstack exec -t check-backup-helper -- tree /var/backup
```
Example of system response:
```
/var/backup
|-- base
|   `-- 2020-09-09_11-35-48
|       |-- backup.stream.gz
|       |-- backup.successful
|       |-- grastate.dat
|       |-- xtrabackup_checkpoints
|       `-- xtrabackup_info
|-- incr
|   `-- 2020-09-09_11-35-48
|       |-- 2020-09-10_01-02-36
|       |-- 2020-09-11_01-02-02
|       |-- 2020-09-12_01-01-54
|       |-- 2020-09-13_01-01-55
|       `-- 2020-09-14_01-01-55
`-- lost+found

10 directories, 5 files
```
If you want to restore the full backup, the name from the example above is 2020-09-09_11-35-48. To restore a specific incremental backup, the name from the example above is 2020-09-09_11-35-48/2020-09-12_01-01-54.

In the example above, the backups will be restored in the following strict order:
1. 2020-09-09_11-35-48 - full backup, path /var/backup/base/2020-09-09_11-35-48
2. 2020-09-10_01-02-36 - incremental backup, path /var/backup/incr/2020-09-09_11-35-48/2020-09-10_01-02-36
3. 2020-09-11_01-02-02 - incremental backup, path /var/backup/incr/2020-09-09_11-35-48/2020-09-11_01-02-02
4. 2020-09-12_01-01-54 - incremental backup, path /var/backup/incr/2020-09-09_11-35-48/2020-09-12_01-01-54

Delete the helper pod:

kubectl -n openstack delete -f check_pod.yaml

Pass the following parameters to the mariadb_resque.py script from the OsDpl object:

Parameter	Type	Default	Description
`--backup-name`	String		Name of a folder with backup in `<BASE_BACKUP>` or `<BASE_BACKUP>/<INCREMENTAL_BACKUP>`.
`--replica-restore-timeout`	Integer	`3600`	Timeout in seconds for 1 replica data to be restored to the `mysql` data directory. Also, includes time for spawning a rescue runner pod in Kubernetes and extracting data from a backup archive.

Edit the OpenStackDeployment object as follows:

spec:
  services:
    database:
      mariadb:
        values:
          manifests:
            job_mariadb_phy_restore: true
          conf:
            phy_restore:
              backup_name: "2020-09-09_11-35-48/2020-09-12_01-01-54"
              replica_restore_timeout: 7200

Wait until the mariadb-phy-restore job suceeds:

kubectl -n openstack get jobs mariadb-phy-restore -o jsonpath='{.status}'

The mariadb-phy-restore job is an immutable object. Therefore, remove the job after each successful execution. To correctly remove the job, clean up all the settings from the OpenStackDeployment object that you have configured during step 7 of this procedure. This will remove all related pods as well.

Important

If mariadb-phy-restore fails, the MariaDB Pods do not start automatically. For example, the failure may occur due to discrepancy between the current and backup versions of MariaDB, broken backup archive, and so on.

Assess the mariadb-phy-restore job log to identify the issue:

kubectl -n openstack logs --tail=10000 -l application=mariadb-phy-restore,job-name=mariadb-phy-restore

If the restoration process does not start due to the MariaDB versions discrepancy:

Use other backup file with the corresponding MariaDB version for restoration, if any.
Start MariaDB Pods without restoration:
```
kubectl scale --replicas=3 sts/mariadb-server -n openstack
```
The command above restores the previous cluster state.