So I will write one of my few tech blog posts. Yes I know, a rarity. Linux clusters are pretty common place these days. redhat clustering is just 1 of the several ways to achieve it. The one thing that I have found out the hardway is always , always check if the cluster you are working on has fencing. If it doesn’t have fencing , set it up immediately. Basically a HA cluster without fencing is the equivalent of not having HA.
fencing is a very simply concept. All it means is that if server A notices server B in the cluster has issues, it can reboot or power it off using a lights out management system (could even be drac) . Even the most simple fencing of shoot-the-other-node-in-the-head gives you a lot of stability.
So lesson learnt setup fencing. Test fencing.