Configure a periodic backup for MariaDB databases

After the MOS deployment, the cluster confguration includes the MariaDB backup functionality. By default, the Kubernetes cron job responsible for the MariaDB backup is in the suspended state.

To enable the MariaDB databases backup:

  1. Enable the backup in the OpenStackDeployment object:

    spec:
      features:
        database:
          backup:
            enabled: true
    
  2. Verify that the mariadb-phy-backup CronJob object is present:

    kubectl -n openstack get cronjob mariadb-phy-backup
    

    Example of a positive system response:

    apiVersion: batch/v1beta1
    kind: CronJob
    metadata:
      annotations:
        openstackhelm.openstack.org/release_uuid: ""
      creationTimestamp: "2020-09-08T14:13:48Z"
      managedFields:
      <<<skipped>>>>
      name: mariadb-phy-backup
      namespace: openstack
      resourceVersion: "726449"
      selfLink: /apis/batch/v1beta1/namespaces/openstack/cronjobs/mariadb-phy-backup
      uid: 88c9be21-a160-4de1-afcf-0853697dd1a1
    spec:
      concurrencyPolicy: Forbid
      failedJobsHistoryLimit: 1
      jobTemplate:
        metadata:
          creationTimestamp: null
          labels:
            application: mariadb-phy-backup
            component: backup
            release_group: openstack-mariadb
        spec:
          activeDeadlineSeconds: 4200
          backoffLimit: 0
          completions: 1
          parallelism: 1
          template:
            metadata:
              creationTimestamp: null
              labels:
                application: mariadb-phy-backup
                component: backup
                release_group: openstack-mariadb
            spec:
              containers:
              - command:
                - /tmp/mariadb_resque.py
                - backup
                - --backup-timeout
                - "3600"
                - --backup-type
                - incremental
                env:
                - name: MARIADB_BACKUPS_TO_KEEP
                  value: "10"
                - name: MARIADB_BACKUP_PVC_NAME
                  value: mariadb-phy-backup-data
                - name: MARIADB_FULL_BACKUP_CYCLE
                  value: "604800"
                - name: MARIADB_REPLICAS
                  value: "3"
                - name: MARIADB_BACKUP_REQUIRED_SPACE_RATIO
                  value: "1.2"
                - name: MARIADB_RESQUE_RUNNER_IMAGE
                  value: docker-dev-kaas-local.docker.mirantis.net/general/mariadb:10.4.14-bionic-20200812025059
                - name: MARIADB_RESQUE_RUNNER_SERVICE_ACCOUNT
                  value: mariadb-phy-backup-runner
                - name: MARIADB_RESQUE_RUNNER_POD_NAME_PREFIX
                  value: openstack-mariadb
                - name: MARIADB_POD_NAMESPACE
                  valueFrom:
                    fieldRef:
                      apiVersion: v1
                      fieldPath: metadata.namespace
                image: docker-dev-kaas-local.docker.mirantis.net/general/mariadb:10.4.14-bionic-20200812025059
                imagePullPolicy: IfNotPresent
                name: phy-backup
                resources: {}
                securityContext:
                  allowPrivilegeEscalation: false
                  readOnlyRootFilesystem: true
                terminationMessagePath: /dev/termination-log
                terminationMessagePolicy: File
                volumeMounts:
                - mountPath: /tmp
                  name: pod-tmp
                - mountPath: /tmp/mariadb_resque.py
                  name: mariadb-bin
                  readOnly: true
                  subPath: mariadb_resque.py
                - mountPath: /tmp/resque_runner.yaml.j2
                  name: mariadb-bin
                  readOnly: true
                  subPath: resque_runner.yaml.j2
                - mountPath: /etc/mysql/admin_user.cnf
                  name: mariadb-secrets
                  readOnly: true
                  subPath: admin_user.cnf
              dnsPolicy: ClusterFirst
              initContainers:
              - command:
                - kubernetes-entrypoint
                env:
                - name: POD_NAME
                  valueFrom:
                    fieldRef:
                      apiVersion: v1
                      fieldPath: metadata.name
                - name: NAMESPACE
                  valueFrom:
                    fieldRef:
                      apiVersion: v1
                      fieldPath: metadata.namespace
                - name: INTERFACE_NAME
                  value: eth0
                - name: PATH
                  value: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/
                - name: DEPENDENCY_SERVICE
                - name: DEPENDENCY_DAEMONSET
                - name: DEPENDENCY_CONTAINER
                - name: DEPENDENCY_POD_JSON
                - name: DEPENDENCY_CUSTOM_RESOURCE
                image: docker-dev-kaas-local.docker.mirantis.net/openstack/extra/kubernetes-entrypoint:v1.0.0-20200311160233
                imagePullPolicy: IfNotPresent
                name: init
                resources: {}
                securityContext:
                  allowPrivilegeEscalation: false
                  readOnlyRootFilesystem: true
                  runAsUser: 65534
                terminationMessagePath: /dev/termination-log
                terminationMessagePolicy: File
              nodeSelector:
                openstack-control-plane: enabled
              restartPolicy: Never
              schedulerName: default-scheduler
              securityContext:
                runAsUser: 999
              serviceAccount: mariadb-phy-backup
              serviceAccountName: mariadb-phy-backup
              terminationGracePeriodSeconds: 30
              volumes:
              - emptyDir: {}
                name: pod-tmp
              - name: mariadb-secrets
                secret:
                  defaultMode: 292
                  secretName: mariadb-secrets
              - configMap:
                  defaultMode: 365
                  name: mariadb-bin
                name: mariadb-bin
      schedule: 0 1 * * *
      successfulJobsHistoryLimit: 3
      suspend: false
    
  3. If required, modify the default backup configuration. By default, the backup is set up as follows:

    • Runs on a daily basis at 01:00 AM

    • Creates incremental backups daily and full backups weekly

    • Keeps 10 latest full backups

    • Saves backups to the mariadb-phy-backup-data PVC

    • The backup timeout is 3600 seconds

    • The backup type is incremental

    As illustrated in the cron job exmample, the mariadb_resque.py script launches backups of the MariaDB Galera cluster. The script accepts settings through parameters and environment variables.

    The following table describes the parameters that you can pass to the cron job and override from the OpenStackDeployment object.

    MariaDB backup: Configuration parameters

    Parameter

    Type

    Default

    Description

    --backup-type

    String

    incremental

    Type of a backup. The list of possible values include:

    • incremental

      If the newest full backup is older than the value of the full_backup_cycle parameter, the system performs a full backup. Otherwise, the system performs an incremental backup of the newest full backup.

    • full

      Always performs only a full backup.

    Usage example:

    spec:
      features:
        database:
          backup:
            backup_type: incremental
    

    --backup-timeout

    Integer

    21600

    Timeout in seconds for the system to wait for the backup operation to succeed.

    Usage example:

    spec:
      services:
        database:
          mariadb:
            values:
              conf:
                phy_backup:
                  backup_timeout: 30000
    

    --allow-unsafe-backup

    Boolean

    false

    If set to true, enables the MariaDB cluster backup in a not fully operational cluster, where:

    • The current number of ready pods is not equal to MARIADB_REPLICAS.

    • Some replicas do not have healthy wsrep statuses.

    Usage example:

    spec:
      services:
        database:
          mariadb:
            values:
              conf:
                phy_backup:
                  allow_unsafe_backup: true
    

    The following table describes the environment variables that you can pass to the cron job and override from the OpenStackDeployment object.

    MariaDB backup: Environment variables

    Variable

    Type

    Default

    Description

    MARIADB_BACKUPS_TO_KEEP

    Integer

    10

    Number of full backups to keep.

    Usage example:

    spec:
      features:
        database:
          backup:
            backups_to_keep: 3
    

    MARIADB_BACKUP_PVC_NAME

    String

    mariadb-phy-backup-data

    Persistent volume claim used to store backups.

    Usage example:

    spec:
      services:
        database:
          mariadb:
            values:
              conf:
                phy_backup:
                  backup_pvc_name: mariadb-phy-backup-data
    

    MARIADB_FULL_BACKUP_CYCLE

    Integer

    604800

    Number of seconds that defines a period between 2 full backups. During this period, incremental backups are performed. The parameter is taken into account only if backup_type is set to incremental. Otherwise, it is ignored. For example, with full_backup_cycle set to 604800 seconds a full backup is taken weekly and, if cron is set to 0 0 * * *, an incremental backup is performed on daily basis.

    Usage example:

    spec:
      features:
        database:
          backup:
            full_backup_cycle: 70000
    

    MARIADB_BACKUP_REQUIRED_SPACE_RATIO

    Floating

    1.2

    Multiplier for the database size to predict the space required to create a backup, either full or incremental, and perform a restoration keeping the uncompressed backup files on the same file system as the compressed ones.

    To estimate the size of MARIADB_BACKUP_REQUIRED_SPACE_RATIO, use the following formula: size of (1 uncompressed full backup + all related incremental uncompressed backups + 1 full compressed backup) in KB =< (DB_SIZE * MARIADB_BACKUP_REQUIRED_SPACE_RATIO) in KB.

    The DB_SIZE is the disk space allocated in the MySQL data directory, which is /var/lib/mysql, for databases data excluding galera.cache and ib_logfile* files. This parameter prevents the backup PVC from being full in the middle of the restoration and backup procedures. If the current available space is lower than DB_SIZE * MARIADB_BACKUP_REQUIRED_SPACE_RATIO, the backup script fails before the system starts the actual backup and the overall status of the backup job is failed.

    Usage example:

    spec:
      services:
        database:
          mariadb:
            values:
              conf:
                phy_backup:
                  backup_required_space_ratio: 1.4
    

    For example, to perform full backups monthly and incremental backups daily at 02:30 AM and keep the backups for the last six months, configure the database backup in your OpenStackDeployment object as follows:

    spec:
      features:
        database:
          backup:
            enabled: true
            backups_to_keep: 6
            schedule_time: '30 2 * * *'
            full_backup_cycle: 2628000