Restart MariaDB Galera cluster After Reboot or Galera cluster recovery.

share on:
Restart Galera cluster after reboot.

Restart Galera Cluster after reboot.

Starting Mariadb Galera after reboot on CentOS 7 is the focus of this tutorial. MariaDB Galera will usually need a manual start after a reboot of the CentOS 7. We will check the whole process of starting MariaDB Galera After the reboot and will also try to automate it.

Now that we have installed MariaDB Galera Cluster, we need to make sure that we can automatically Restart Galera cluster after reboot. if you still haven’t installed MariaDB Galera on CentOS 7 then you can follow this tutorial.

How to Setup MariaDB Galera Cluster on CentOS 7.

MariaDB version

[[email protected] ~]# mysql -V
mysql Ver 15.1 Distrib 10.0.29-MariaDB, for Linux (x86_64) using readline 5.1

 

[[email protected] ~]# mysql -V
mysql Ver 15.1 Distrib 10.0.29-MariaDB, for Linux (x86_64) using readline 5.1

 

[[email protected] ~]# mysql -V
mysql Ver 15.1 Distrib 10.0.29-MariaDB, for Linux (x86_64) using readline 5.1

 

Errors when we Restart MariaDB Galera after Reboot.

When we Restart Galera cluster after reboot, it may not start and can show these errors. (did on my servers)

[[email protected] ~]# service mysql start --wsrep-new-cluster
Starting MySQL.170126 10:28:58 mysqld_safe Logging to '/var/lib/mysql/galera-01.engy.remote.err'.
170126 10:28:58 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
.. ERROR!

 

Let’s take a look at these errors and try to resolve them. The errors are being logged in /var/lib/mysql/galera-01.engy.remote.err file on galera-01 server.

[[email protected] ~]# tail -f /var/lib/mysql/galera-01.engy.remote.err
170126 10:29:01 [Note] WSREP: Setting initial position to e49a3e4d-e3ae-11e6-8d23-32480d0156a6:2
170126 10:29:01 [ERROR] WSREP: It may not be safe to bootstrap the cluster from this node. It was not the last one to leave the cluster and may not contain all the updates. To force cluster bootstrap with this node, edit the grastate.dat file manually and set safe_to_bootstrap to 1 .
170126 10:29:01 [ERROR] WSREP: wsrep::connect(gcomm://10.136.42.199,10.136.42.249,10.136.43.4) failed: 7
170126 10:29:01 [ERROR] Aborting

170126 10:29:01 [Note] WSREP: Service disconnected.
170126 10:29:02 [Note] WSREP: Some threads may fail to exit.
170126 10:29:02 [Note] /usr/sbin/mysqld: Shutdown complete

170126 10:29:02 mysqld_safe mysqld from pid file /var/lib/mysql/galera-01.engy.remote.pid ended

 

As suggested we will edit the garstate.dat file and make the necessary changes.

The original contents of garstate.dat file.

[[email protected] ~]# vi /var/lib/mysql/grastate.dat
# GALERA saved state
version: 2.1
uuid: e49a3e4d-e3ae-11e6-8d23-32480d0156a6
seqno: -1
safe_to_bootstrap: 0

 

change safe_to_bootstrap: 1

[[email protected] ~]# vi /var/lib/mysql/grastate.dat
# GALERA saved state
version: 2.1
uuid: e49a3e4d-e3ae-11e6-8d23-32480d0156a6
seqno: -1
safe_to_bootstrap: 1

 

Now try again to restart the main node.

Use the command: service mysql start –wsrep-new-cluster

[[email protected] ~]# sudo /etc/init.d/mysql start --wsrep-new-cluster
Starting MySQL.170126 10:42:22 mysqld_safe Logging to '/var/lib/mysql/galera-01.engy.remote.err'.
170126 10:42:22 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
. SUCCESS!
[[email protected] ~]#

 

On the other nodes, start the Mariadb service.

[[email protected] ~]# service mysql start
[[email protected] ~]# service mysql start

 

Automate the whole process By Cron job.

I have used this cron job to automatically restart the Mariadb Galera after reboot and it is working fine. (for now). I have tested it 4-5 times and haven’t faced any issues yet.

On the main Mariadb Galera node, Set a cron job as follows.
As root. Do verify the location of the garstate.dat file before using this step as a copy paste.

shell> crontab -e
@reboot sed -ie '/safe_to/c\safe_to_bootstarp: 1' /var/lib/mysql/grastate.dat && /etc/init.d/mysql start --wsrep-new-cluster

 

Again Please verify the location of the garstate.dat file file before setting this job as it may be located somewhere else on your system. 

 

On other nodes set a simple Cron job to restart the mariadb service.

shell> crontab -e
@reboot service mysql start

 

Now it will automatically Restart Galera cluster after reboot. Thanks for visiting this page and if you have a better solution for this problem then please do let us know via comments or message on any platform.

Have a Great day. 🙂

Reference:

 

 

share on:
engy

engy

Hello there, My name is Rishi Guleria and I work as a Linux system administrator. I have created this blog to share what I have learned so far and to learn new things. Good Day. :)

Leave a Response

share on: