Resolve cooperative monitoring deadlocks

MaxScale 2.5 introduced cooperative monitoring as a way for two MaxScale instances to decide on their own which is the active and which is the passive instance. In the KB this is now left as the sole option (articles on keepalived, Lsyncd etc. have been removed). Unfortunately, cooperative monitoring does not work well when the system has (or, depending on the monitoring mode, ends up having) an even number of (active) nodes; this may easily lead to the situation when both MaxScale instances become passive and here is none to perform the HA function of a failover, should such be needed.

Example: One primary, three slaves; MaxScales running with "majority of all"; each MaxScale obtains two locks and remains passive due to lack of quorum. Similar scenarios can be created for "majority of running" mode also.

We see at least two ways of resolving such deadlocks:

• With MaxScale-to-MaxScale communication: a strategy like "lowest IP address wins" may help a MaxScale become active when no MaxScale holds a majority of the necessary locks.

• Without MaxScale-to-MaxScale communication: assigning a weight to each locked node and using these weights to form the quorum. Following the above example, if one of the nodes is given a weight of 2, sum of all will not be 4 but 5, so one of the MaxScale instances will have 3/5 of the lock's weights and will thus stay active even though it will only have locks of 1/2 of the nodes (the other MaxScale will also have locks for 1/2 of the nodes, but will only have 2/5 of the sum of all weights).