Change the storage node for MariaDB on bare metal and vSphere clusters

The default storage class cannot be used on a bare metal or vSphere management cluster, so a specially created one is used for this purpose. For storage, this class uses local volumes, which are managed by local-volume-provisioner.

Each node of a management cluster contains a local volume, and the volume bound with a PVC is selected when the backup gets created for the first time. This volume is used for all subsequent backups. Therefore, to ensure reliable backup storage, consider creating a regular backup copy of this volume in a separate location.

If the node that contains backup data must be redeployed, first move the MySQL backup data to another node and update the PVC binding along with the MariaDB backup job to use another node as described below.

Identify a node where backup data is stored

  1. Download and save the following script on the node where kubectl is installed and configured to use the Kubernetes API:

    get_lv_info.sh

  2. Make the script executable and execute it:

    vim get_lv_info.sh
    
    chmod +x get_lv_info.sh
    
    ./get_lv_info.sh
    

    The script outputs the following information:

    Primary local volume

    Current active local volume, which is bound to the PVC using the backup_pvc_name field and which is used to store backup data.

    Secondary local volume

    Unused volumes of two remaining nodes of the management cluster.

    Example of system response:

    Primary local volume
    ====================
    Volume: local-pv-a1c9425b
    Volume path: /mnt/local-volumes/iam/kaas-iam-backup/vol00
    Data PVC: mysql-data-mariadb-server-1
    Backup PVC: mariadb-phy-backup-data
    Node: kaas-node-788dba0a-f931-45ff-a66d-1b583851c3ba
    Machine: master-1
    Internal IP: 10.100.91.50
    
    Secondary local volume
    ----------------------
    Volume: local-pv-8519d270
    Volume path: /mnt/local-volumes/iam/kaas-iam-backup/vol00
    Data PVC: mysql-data-mariadb-server-0
    Node: kaas-node-2b83025a-d4d1-4ccc-a263-11b07150f302
    Machine: master-0
    Internal IP: 10.100.91.51
    
    Secondary local volume
    ----------------------
    Volume: local-pv-1bfef721
    Volume path: /mnt/local-volumes/iam/kaas-iam-backup/vol00
    Data PVC: mysql-data-mariadb-server-2
    Node: kaas-node-f4742907-5fb0-41fb-ba6c-3ce467779754
    Machine: master-2
    Internal IP: 10.100.91.52
    

    Note

    The order of nodes that contain Secondary local volume is random.

  3. Capture details of the node containing the primary local volume for further usage. For example, you can use the Internal IP value to SSH to the required node and copy the backup data located under Volume path to a separate location.

Change the default storage node for MariaDB backups

  1. Capture details of the local volume and node containing backups as described in Identify a node where backup data is stored. Also, capture details of Secondary local volume that you select to move backup data to.

  2. Using Internal IP of Primary local volume, SSH to the corresponding node and create a backup tarball:

    Note

    In the command below, replace <newVolumePath> with the value of the Volume path field of the selected Secondary local volume.

    sudo tar -czPvf ~/mariadb-backup.tar.gz -C <newVolumePath>
    
  3. Using Internal IP of Secondary local volume, SSH to the corresponding node and copy the created backup mariadb-backup.tar.gz using a dedicated utility such as scp, rsync, or other.

  4. Restore mariadb-backup.tar.gz under the selected Volume path:

    sudo tar -xzPvf ~/mariadb-backup.tar.gz -C <newVolumePath>
    
  5. Update the CronJob object to associate it with the new backup node:

    1. Download and save the following helper script on a node where kubectl is installed and configured to use Kubernetes API:

      fix_cronjob_pvc

    2. Make the script executable:

      vim fix_cronjob_pvc.sh
      
      chmod +x fix_cronjob_pvc.sh
      
    3. Using the Data PVC value of the selected Secondary local volume, run the script:

      ./fix_cronjob_pvc.sh <secondaryDataPVCName>