Periodic OpenStack database backups¶

MOSK uses the Mariabackup utility to back up the MariaDB Galera cluster data where the OpenStack data is stored. The Mariabackup gets launched on a periodic basis as a part of the Kubernetes CronJob included in any MOSK deployment and is suspended by default.

Note

If you are using the default back end to store the backup data, which is Ceph, you can increase the default size of a backup volume. However, make sure to configure the volume size before you deploy OpenStack.

For the default sizes and configuration details, refer to Size of a backup storage.

Enabling the periodic backup¶

MOSK enables you to configure the periodic backup of the OpenStack database through the OpenStackDeployment object. To enable the backup, use the following structure:

spec:
  features:
    database:
      backup:
        enabled: true

By default, the backup job:

Runs backup on a daily basis at 01:00 AM
Creates incremental backups daily and full backups weekly
Keeps 10 latest full backups
Stores backups in the mariadb-phy-backup-data PVC
Has the backup timeout of 3600 seconds
Has the incremental backup type

To verify the configuration of the mariadb-phy-backup CronJob object, run:

kubectl -n openstack get cronjob mariadb-phy-backup

Example of a mariadb-phy-backup CronJob object

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  annotations:
    openstackhelm.openstack.org/release_uuid: ""
  creationTimestamp: "2020-09-08T14:13:48Z"
  managedFields:
  <<<skipped>>>>
  name: mariadb-phy-backup
  namespace: openstack
  resourceVersion: "726449"
  selfLink: /apis/batch/v1beta1/namespaces/openstack/cronjobs/mariadb-phy-backup
  uid: 88c9be21-a160-4de1-afcf-0853697dd1a1
spec:
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 1
  jobTemplate:
    metadata:
      creationTimestamp: null
      labels:
        application: mariadb-phy-backup
        component: backup
        release_group: openstack-mariadb
    spec:
      activeDeadlineSeconds: 4200
      backoffLimit: 0
      completions: 1
      parallelism: 1
      template:
        metadata:
          creationTimestamp: null
          labels:
            application: mariadb-phy-backup
            component: backup
            release_group: openstack-mariadb
        spec:
          containers:
          - command:
            - /tmp/mariadb_resque.py
            - backup
            - --backup-timeout
            - "3600"
            - --backup-type
            - incremental
            env:
            - name: MARIADB_BACKUPS_TO_KEEP
              value: "10"
            - name: MARIADB_BACKUP_PVC_NAME
              value: mariadb-phy-backup-data
            - name: MARIADB_FULL_BACKUP_CYCLE
              value: "604800"
            - name: MARIADB_REPLICAS
              value: "3"
            - name: MARIADB_BACKUP_REQUIRED_SPACE_RATIO
              value: "1.2"
            - name: MARIADB_RESQUE_RUNNER_IMAGE
              value: docker-dev-kaas-local.docker.mirantis.net/general/mariadb:10.4.14-bionic-20200812025059
            - name: MARIADB_RESQUE_RUNNER_SERVICE_ACCOUNT
              value: mariadb-phy-backup-runner
            - name: MARIADB_RESQUE_RUNNER_POD_NAME_PREFIX
              value: openstack-mariadb
            - name: MARIADB_POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            image: docker-dev-kaas-local.docker.mirantis.net/general/mariadb:10.4.14-bionic-20200812025059
            imagePullPolicy: IfNotPresent
            name: phy-backup
            resources: {}
            securityContext:
              allowPrivilegeEscalation: false
              readOnlyRootFilesystem: true
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            volumeMounts:
            - mountPath: /tmp
              name: pod-tmp
            - mountPath: /tmp/mariadb_resque.py
              name: mariadb-bin
              readOnly: true
              subPath: mariadb_resque.py
            - mountPath: /tmp/resque_runner.yaml.j2
              name: mariadb-bin
              readOnly: true
              subPath: resque_runner.yaml.j2
            - mountPath: /etc/mysql/admin_user.cnf
              name: mariadb-secrets
              readOnly: true
              subPath: admin_user.cnf
          dnsPolicy: ClusterFirst
          initContainers:
          - command:
            - kubernetes-entrypoint
            env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: INTERFACE_NAME
              value: eth0
            - name: PATH
              value: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/
            - name: DEPENDENCY_SERVICE
            - name: DEPENDENCY_DAEMONSET
            - name: DEPENDENCY_CONTAINER
            - name: DEPENDENCY_POD_JSON
            - name: DEPENDENCY_CUSTOM_RESOURCE
            image: docker-dev-kaas-local.docker.mirantis.net/openstack/extra/kubernetes-entrypoint:v1.0.0-20200311160233
            imagePullPolicy: IfNotPresent
            name: init
            resources: {}
            securityContext:
              allowPrivilegeEscalation: false
              readOnlyRootFilesystem: true
              runAsUser: 65534
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
          nodeSelector:
            openstack-control-plane: enabled
          restartPolicy: Never
          schedulerName: default-scheduler
          securityContext:
            runAsUser: 999
          serviceAccount: mariadb-phy-backup
          serviceAccountName: mariadb-phy-backup
          terminationGracePeriodSeconds: 30
          volumes:
          - emptyDir: {}
            name: pod-tmp
          - name: mariadb-secrets
            secret:
              defaultMode: 292
              secretName: mariadb-secrets
          - configMap:
              defaultMode: 365
              name: mariadb-bin
            name: mariadb-bin
  schedule: 0 1 * * *
  successfulJobsHistoryLimit: 3
  suspend: false

Overriding the default configuration¶

To override the default configuration, set the parameters and environment variables that are passed to the CronJob as described in the tables below.

MariaDB backup: Configuration parameters¶
Parameter	Type	Default	Description
`--backup-type`	String	`incremental`	Type of a backup. The list of possible values include: `incremental` If the newest full backup is older than the value of the `full_backup_cycle` parameter, the system performs a full backup. Otherwise, the system performs an incremental backup of the newest full backup. `full` Always performs only a full backup. Usage example: spec: features: database: backup: backup_type: incremental
`--backup-timeout`	Integer	`21600`	Timeout in seconds for the system to wait for the backup operation to succeed. Usage example: spec: services: database: mariadb: values: conf: phy_backup: backup_timeout: 30000
`--allow-unsafe-backup`	Boolean	`false`	Not recommended, removed since MOSK 22.4. If set to `true`, enables the MariaDB cluster backup in a not fully operational cluster, where: The current number of ready pods is not equal to `MARIADB_REPLICAS`. Some replicas do not have healthy wsrep statuses. Usage example: spec: services: database: mariadb: values: conf: phy_backup: allow_unsafe_backup: true

MariaDB backup: Environment variables¶
Variable	Type	Default	Description
`MARIADB_BACKUPS_TO_KEEP`	Integer	`10`	Number of full backups to keep. Usage example: spec: features: database: backup: backups_to_keep: 3
`MARIADB_BACKUP_PVC_NAME`	String	`mariadb-phy-backup-data`	Persistent volume claim used to store backups. Usage example: spec: services: database: mariadb: values: conf: phy_backup: backup_pvc_name: mariadb-phy-backup-data
`MARIADB_FULL_BACKUP_CYCLE`	Integer	`604800`	Number of seconds that defines a period between 2 full backups. During this period, incremental backups are performed. The parameter is taken into account only if `backup_type` is set to `incremental`. Otherwise, it is ignored. For example, with `full_backup_cycle` set to `604800` seconds a full backup is taken weekly and, if cron is set to `0 0 * * `, an incremental backup is performed on daily basis. Usage example:* spec: features: database: backup: full_backup_cycle: 70000
`MARIADB_BACKUP_REQUIRED_SPACE_RATIO`	Floating	`1.2`	Multiplier for the database size to predict the space required to create a backup, either full or incremental, and perform a restoration keeping the uncompressed backup files on the same file system as the compressed ones. To estimate the size of `MARIADB_BACKUP_REQUIRED_SPACE_RATIO`, use the following formula: size of (1 uncompressed full backup + all related incremental uncompressed backups + 1 full compressed backup) in KB =< (`DB_SIZE` * `MARIADB_BACKUP_REQUIRED_SPACE_RATIO`) in KB. The `DB_SIZE` is the disk space allocated in the MySQL data directory, which is `/var/lib/mysql`, for databases data excluding `galera.cache` and `ib_logfile` files. This parameter prevents the backup PVC from being full in the middle of the restoration and backup procedures. If the current available space is lower than `DB_SIZE` `MARIADB_BACKUP_REQUIRED_SPACE_RATIO`, the backup script fails before the system starts the actual backup and the overall status of the backup job is failed. Usage example: spec: services: database: mariadb: values: conf: phy_backup: backup_required_space_ratio: 1.4

For example, to perform full backups monthly and incremental backups daily at 02:30 AM and keep the backups for the last six months, configure the database backup in your OpenStackDeployment object as follows:

spec:
  features:
    database:
      backup:
        enabled: true
        backups_to_keep: 6
        schedule_time: '30 2 * * *'
        full_backup_cycle: 2628000