As the manual procedure of the Galera cluster restoration is prone to operator errors, we recommend restoring the cluster using the Jenkins pipeline as described in Restore a Galera cluster and database automatically. Though, if you cannot apply the automatic procedure to your deployment for some reason, use the manual instruction included in this section.
Note
To restore a Galera database using the Jenkins pipeline, see Restore a Galera cluster and database automatically.
To restore a Galera cluster manually:
On all Galera dbs
nodes:
Stop all the MySQL processes.
Verify that the MySQL processes are stopped:
ps aux | grep mysql
Identify the last shutdown Galera node:
/var/lib/mysql/grastate.dat
file on every Galera node,
compare the seqno
value. The Galera node that contains
the maximum seqno
value is the last shutdown node.seqno
value is equal on all three nodes, identify the
node on which the /var/lib/mysql/gvwstate.dat
file exists.
The Galera node that contains this file is the last shutdown node.Remove the grastate.dat
and ib_logfiles
:
rm /var/lib/mysql/grastate.dat
rm /var/lib/mysql/ib_logfile*
Log in to the last shutdown Galera node.
In /etc/mysql/my.cnf
:
Comment out the wsrep_cluster_address
line:
...
#wsrep_cluster_address="gcomm://192.168.0.1,192.168.0.2,192.168.0.3"
...
Add the wsrep_cluster_address
parameter without any IP address
specified.
...
wsrep_cluster_address=gcomm://
...
Start MySQL:
service mysql start
Validate the current status of the Galera cluster:
salt-call mysql.status | grep -A1 wsrep_cluster_size
Start MySQL on the second Galera node. Wait until the node re-joins the cluster.
When the cluster size equals to two, start MySQL on the third node.
Verify the MySQL status. The cluster size should be equal to three:
salt-call mysql.status | grep -A1 wsrep_cluster_size
Log in to the last shutdown Galera node.
In /etc/mysql/my.cnf
:
Uncomment the line with the wsrep_cluster_address
parameter:
...
wsrep_cluster_address="gcomm://192.168.0.1,192.168.0.2,192.168.0.3"
...
Remove the wsrep_cluster_address
parameter without any IP address
specified.
Verify that the /etc/salt/.galera_bootstrap
file exists on every
dbs
node. Otherwise, create one:
touch /etc/salt/.galera_bootstrap
rw-rr- 1 root root 0 Feb 12 08:31 /etc/salt/.galera_bootstrap
Revert the changes made in the cluster/openstack/database/init.yml
file in the step 2 during Prepare for a Galera cluster restoration.