Restore MariaDB databases¶

During the restore procedure, the MariaDB service will be unavailable because the MariaDB StatefulSet scales down to 0 replicas. Therefore, plan the maintenance window according to the database size. The restore speed depends on the following:

Network throughput
Storage performance where backups are kept
Local disks performance of nodes with MariaDB local volumes

To restore MariaDB databases:

Obtain an image of the MariaDB container:

kubectl -n kaas get pods mariadb-server-0 -o jsonpath='{.spec.containers[0].image}'

Create the check_pod.yaml file to create the helper pod required to view the backup volume content. For example:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: check-backup-helper
  namespace: kaas
---
apiVersion: v1
kind: Pod
metadata:
  name: check-backup-helper
  namespace: kaas
  labels:
    application: check-backup-helper
spec:
  containers:
    - name: helper
      securityContext:
        allowPrivilegeEscalation: false
        runAsUser: 0
        readOnlyRootFilesystem: true
      command:
        - sleep
        - infinity
      image: <<insert_image_of_mariadb_container_here>>
      imagePullPolicy: IfNotPresent
      volumeMounts:
        - name: pod-tmp
          mountPath: /tmp
        - mountPath: /var/backup
          name: mysql-backup
  restartPolicy: Never
  serviceAccount: check-backup-helper
  serviceAccountName: check-backup-helper
  volumes:
    - name: pod-tmp
      emptyDir: {}
    - name: mariadb-secrets
      secret:
        secretName: mariadb-secrets
        defaultMode: 0444
    - name: mariadb-bin
      configMap:
        name: mariadb-bin
        defaultMode: 0555
    - name: mysql-backup
      persistentVolumeClaim:
        claimName: mariadb-phy-backup-data

Create the helper pod:

kubectl -n kaas apply -f check_pod.yaml

Obtain the name of the backup to restore:
```
kubectl -n kaas exec -t check-backup-helper -- tree /var/backup
```
Example of system response:
```
/var/backup
|-- base
|   `-- 2021-09-09_11-35-48
|       |-- backup.stream.gz
|       |-- backup.successful
|       |-- grastate.dat
|       |-- xtrabackup_checkpoints
|       `-- xtrabackup_info
|-- incr
|   `-- 2021-09-09_11-35-48
|       |-- 2021-09-10_01-02-36
|       |-- 2021-09-11_01-02-02
|       |-- 2021-09-12_01-01-54
|       |-- 2021-09-13_01-01-55
|       `-- 2021-09-14_01-01-55
`-- lost+found

10 directories, 5 files
```
If you want to restore the full backup, the name from the example above is 2021-09-09_11-35-48. To restore a specific incremental backup, the name from the example above is 2021-09-09_11-35-48/2021-09-12_01-01-54.

In the example above, the backups will be restored in the following strict order:
1. 2021-09-09_11-35-48 - full backup, path /var/backup/base/2021-09-09_11-35-48
2. 2021-09-10_01-02-36 - incremental backup, path /var/backup/incr/2021-09-09_11-35-48/2021-09-10_01-02-36
3. 2021-09-11_01-02-02 - incremental backup, path /var/backup/incr/2021-09-09_11-35-48/2021-09-11_01-02-02
4. 2021-09-12_01-01-54 - incremental backup, path /var/backup/incr/2021-09-09_11-35-48/2021-09-12_01-01-54
Delete the helper pod to prevent PVC multi-attach issues:
```
kubectl -n kaas delete -f check_pod.yaml
```

Verify that no other restore job exists on the cluster:

cd kaas-bootstrap

kubectl -n kaas get jobs | grep restore

kubectl -n kaas get po | grep check-backup-helper

Edit the Cluster object by configuring the MariaDB parameters. For example:

spec:
  providerSpec:
    kaas:
      management:
        helmReleases:
        - name: iam
          values:
            keycloak:
              mariadb:
                manifests:
                  job_mariadb_phy_restore: true
                conf:
                  phy_restore:
                    backup_name: "2021-09-09_11-35-48/2021-09-12_01-01-54"
                    replica_restore_timeout: 7200

Parameter	Type	Default	Description
`backup-name`	String	-	Name of a folder with backup in `<baseBackup>` or `<baseBackup>/<incrementalBackup>`.
`replica-restore-timeout`	Integer	`3600`	Timeout in seconds for 1 replica data to be restored to the `mysql` data directory. Also, includes time for spawning a rescue runner pod in Kubernetes and extracting data from a backup archive.

Wait until the mariadb-phy-restore job succeeds:

kubectl -n kaas get jobs mariadb-phy-restore -o jsonpath='{.status}'

The mariadb-phy-restore job is an immutable object. Therefore, remove the job after each execution. To correctly remove the job, clean up all settings from the Cluster object that you have configured during step 7 of this procedure. This will remove all related pods as well.

Note

If you create a new user after creating the MariaDB backup file, such user obviously will not exist in the database after restoring MariaDB. But Keycloak may still contain cache about such user. Therefore, during an attempt of this user to log in, the Container Cloud web UI may start the authentication procedure that fails with the following error: Data loading failed: Failed to log in: Failed to get token. Reason: “User not found”. To clear cache in Keycloak, refer to the official Keycloak documentation.