repmgr/docs/repmgrd-failover-mechanism.md

repmgrd's failover algorithm

When implementing automatic failover, there are two factors which are critical in ensuring the desired result is achieved:

• has the master node genuinely failed?
• which is the best node to promote to the new master?

This document outlines repmgrd's decision-making process during automatic failover for standbys directly connected to the master node.

Master node failure detection

If a repmgrd instance running on a PostgreSQL standby node is unable to connect to the master node, this doesn't neccesarily mean that the master is down and a failover is required. Factors such as network connectivity issues could mean that even though the standby node is isolated, the replication cluster as a whole is functioning correctly, and promoting the standby without further verification could result in a "split-brain" situation.

In the event that repmgrd is unable to connect to the master node, it will attempt to reconnect to the master server several times (as defined by the reconnect_attempts parameter in repmgr.conf), with reconnection attempts occuring at the interval specified by reconnect_interval. This happens to verify that the master is definitively not accessible (e.g. that connection was not lost due to a brief network glitch).

Appropriate values for these settings will depend very much on the replication cluster environment. There will necessarily be a trade-off between the time it takes to assume the master is not reachable, and the reliability of that conclusion. A standby in a different physical location to the master will probably need a longer check interval to rule out possible network issues, whereas one located in the same rack with a direct connection between servers could perform the check very quickly.

Note that it's possible the master comes back online after this point is reached, but before a new master has been selected; in this case it will be noticed during the selection of a new master and no actual failover will take place.

Promotion candidate selection

Once repmgrd has decided the master is definitively unreachable, following checks will be carried out:

• attempts to connect to all other nodes in the cluster (including the witness node, if defined) to establish the state of the cluster, including their current LSN

• If less than half of the nodes are visible (from the viewpoint of this node), repmgrd will not take any further action. This is to ensure that e.g. if a replication cluster is spread over multiple data centres, a split-brain situation does not occur if there is a network failure between datacentres. Note that if nodes are split evenly between data centres, a witness server can be used to establish the "majority" daat centre.

• repmgrd polls all visible servers and waits for each node to return a valid LSN; it updates the LSN previously stored for this node if it has increased since the initial check

• once all LSNs have been retrieved, repmgrd will check for the highest LSN; if its own node has the highest LSN, it will attempt to promote itself (using the command defined in promote_command in repmgr.conf. Note that if using repmgr standby promote as the promotion command, and the original master becomes available before the promotion takes effect, repmgr will return an error and no promotion will take place, and repmgrd will resume monitoring as usual.

• if the node is not the promotion candidate, repmgrd will execute the follow_command defined in repmgr.conf. If using repmgr standby follow here, repmgr will attempt to detect the new master node and attach to that.