Restore a Galera cluster manually¶

As the manual procedure of the Galera cluster restoration is prone to operator errors, we recommend restoring the cluster using the Jenkins pipeline as described in Restore a Galera cluster and database automatically. Though, if you cannot apply the automatic procedure to your deployment for some reason, use the manual instruction included in this section.

To restore a Galera cluster manually:

1. On all Galera dbs nodes:

   1. Stop all the MySQL processes.

   2. Verify that the MySQL processes are stopped:

      ps aux | grep mysql
      

   3. Identify the last shutdown Galera node:

      1. In the /var/lib/mysql/grastate.dat file on every Galera node, compare the seqno value. The Galera node that contains the maximum seqno value is the last shutdown node.
      2. If the seqno value is equal on all three nodes, identify the node on which the /var/lib/mysql/gvwstate.dat file exists. The Galera node that contains this file is the last shutdown node.

   4. Remove the grastate.dat and ib_logfiles:

      rm /var/lib/mysql/grastate.dat
      rm /var/lib/mysql/ib_logfile*
      

2. Log in to the last shutdown Galera node.

3. In /etc/mysql/my.cnf:

   1. Comment out the wsrep_cluster_address line:

      ...
      #wsrep_cluster_address="gcomm://192.168.0.1,192.168.0.2,192.168.0.3"
      ...
      

   2. Add the wsrep_cluster_address parameter without any IP address specified.

      ...
      wsrep_cluster_address=gcomm://
      ...
      

4. Start MySQL:

   service mysql start
   

5. Validate the current status of the Galera cluster:

   salt-call mysql.status | grep -A1 wsrep_cluster_size
   

6. Start MySQL on the second Galera node. Wait until the node re-joins the cluster.

7. When the cluster size equals to two, start MySQL on the third node.

8. Verify the MySQL status. The cluster size should be equal to three:

   salt-call mysql.status | grep -A1 wsrep_cluster_size
   

9. Log in to the last shutdown Galera node.

10. In /etc/mysql/my.cnf:

    1. Uncomment the line with the wsrep_cluster_address parameter:

       ...
       wsrep_cluster_address="gcomm://192.168.0.1,192.168.0.2,192.168.0.3"
       ...
       

    2. Remove the wsrep_cluster_address parameter without any IP address specified.

11. Verify that the /etc/salt/.galera_bootstrap file exists on every dbs node. Otherwise, create one:

    touch /etc/salt/.galera_bootstrap
    rw-rr- 1 root root 0 Feb 12 08:31 /etc/salt/.galera_bootstrap
    

12. Revert the changes made in the cluster/openstack/database/init.yml file in the step 2 during Prepare for a Galera cluster restoration.