A Shell Script to keep the Datanodes on MySQL NDB Clusters live and stable
When i installed an ndb clusters, i found sometimes one of the datanodes, fails for some reason and they never start again untill it is manually done. So i wrote a shell script for the datanodes to check itself. The script checks in the ndbd process is running, if not its starts the process and writes a log. I added this shell script to the cron job to run the script every 10 minutes. Now the ndb clusters is perfectly stable for some months.
You may ask, 10 mins is too long. Does it mean, the database server will be down for 10 mins. Not beacuse there are more than 1 data node in the cluster, during this 10 mins other nodes will serve.
#!/bin/sh
logfile=/usr/local/mysql-cluster/restart.log
PID=`ps -eo ‘tty pid args’ | grep ‘ndbd’ | grep -v grep | tr -s ” | cut -f2 -d ”`
if [ -z "$PID" ]
then
message=`service ndbd start`
echo “`date` – NDBD started”>>$logfile
fi
Cron job that I added, (edit the file /etc/crontab and add the following line at the end)
*/10 * * * * root /path/to/the/script
This would keep the ndbclusters live for ever.
