Restore MariaDB databases

During the restore procedure, the MariaDB service will be unavailable because the MariaDB StatefulSet scales down to 0 replicas. Therefore, plan the maintenance window according to the database size. The restore speed depends on the following:

  • Network throughput

  • Storage performance where backups are kept

  • Local disks performance of nodes with MariaDB local volumes

To restore MariaDB databases:

  1. Obtain an image of the MariaDB container:

    kubectl -n kaas get pods mariadb-server-0 -o jsonpath='{.spec.containers[0].image}'
    
  2. Create the check_pod.yaml file to create the helper pod required to view the backup volume content. For example:

    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: check-backup-helper
      namespace: kaas
    ---
    apiVersion: v1
    kind: Pod
    metadata:
      name: check-backup-helper
      namespace: kaas
      labels:
        application: check-backup-helper
    spec:
      containers:
        - name: helper
          securityContext:
            allowPrivilegeEscalation: false
            runAsUser: 0
            readOnlyRootFilesystem: true
          command:
            - sleep
            - infinity
          image: <<insert_image_of_mariadb_container_here>>
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: pod-tmp
              mountPath: /tmp
            - mountPath: /var/backup
              name: mysql-backup
      restartPolicy: Never
      serviceAccount: check-backup-helper
      serviceAccountName: check-backup-helper
      volumes:
        - name: pod-tmp
          emptyDir: {}
        - name: mariadb-secrets
          secret:
            secretName: mariadb-secrets
            defaultMode: 0444
        - name: mariadb-bin
          configMap:
            name: mariadb-bin
            defaultMode: 0555
        - name: mysql-backup
          persistentVolumeClaim:
            claimName: mariadb-phy-backup-data
    
  3. Create the helper pod:

    kubectl -n kaas apply -f check_pod.yaml
    
  4. Obtain the name of the backup to restore:

    kubectl -n kaas exec -t check-backup-helper -- tree /var/backup
    

    Example of system response:

    /var/backup
    |-- base
    |   `-- 2021-09-09_11-35-48
    |       |-- backup.stream.gz
    |       |-- backup.successful
    |       |-- grastate.dat
    |       |-- xtrabackup_checkpoints
    |       `-- xtrabackup_info
    |-- incr
    |   `-- 2021-09-09_11-35-48
    |       |-- 2021-09-10_01-02-36
    |       |-- 2021-09-11_01-02-02
    |       |-- 2021-09-12_01-01-54
    |       |-- 2021-09-13_01-01-55
    |       `-- 2021-09-14_01-01-55
    `-- lost+found
    
    10 directories, 5 files
    

    If you want to restore the full backup, the name from the example above is 2021-09-09_11-35-48. To restore a specific incremental backup, the name from the example above is 2021-09-09_11-35-48/2021-09-12_01-01-54.

    In the example above, the backups will be restored in the following strict order:

    1. 2021-09-09_11-35-48 - full backup, path /var/backup/base/2021-09-09_11-35-48

    2. 2021-09-10_01-02-36 - incremental backup, path /var/backup/incr/2021-09-09_11-35-48/2021-09-10_01-02-36

    3. 2021-09-11_01-02-02 - incremental backup, path /var/backup/incr/2021-09-09_11-35-48/2021-09-11_01-02-02

    4. 2021-09-12_01-01-54 - incremental backup, path /var/backup/incr/2021-09-09_11-35-48/2021-09-12_01-01-54

  5. Delete the helper pod to prevent PVC multi-attach issues:

    kubectl -n kaas delete -f check_pod.yaml
    
  6. Verify that no other restore job exists on the cluster:

    cd kaas-bootstrap
    
    kubectl -n kaas get jobs | grep restore
    
    kubectl -n kaas get po | grep check-backup-helper
    
  7. Edit the Cluster object by configuring the MariaDB parameters. For example:

    spec:
      services:
        database:
          mariadb:
            values:
              manifests:
                job_mariadb_phy_restore: true
              conf:
                phy_restore:
                  backup_name: "2021-09-09_11-35-48/2021-09-12_01-01-54"
                  replica_restore_timeout: 7200
    

    Parameter

    Type

    Default

    Description

    backup-name

    String

    -

    Name of a folder with backup in <baseBackup> or <baseBackup>/<incrementalBackup>.

    replica-restore-timeout

    Integer

    3600

    Timeout in seconds for 1 replica data to be restored to the mysql data directory. Also, includes time for spawning a rescue runner pod in Kubernetes and extracting data from a backup archive.

  8. Wait until the mariadb-phy-restore job succeeds:

    kubectl -n kaas get jobs mariadb-phy-restore -o jsonpath='{.status}'
    
  9. The mariadb-phy-restore job is an immutable object. Therefore, remove the job after each execution. To correctly remove the job, clean up all settings from the Cluster object that you have configured during step 7 of this procedure. This will remove all related pods as well.

Note

If you create a new user after creating the MariaDB backup file, such user obviously will not exist in the database after restoring MariaDB. But Keycloak may still contain cache about such user. Therefore, during an attempt of this user to log in, the Container Cloud web UI may start the authentication procedure that fails with the following error: Data loading failed: Failed to log in: Failed to get token. Reason: “User not found”. To clear cache in Keycloak, refer to the official Keycloak documentation.