diff --git a/doc/configuration.sgml b/doc/configuration.sgml
index ffd4e53e..b9cbb68b 100644
--- a/doc/configuration.sgml
+++ b/doc/configuration.sgml
@@ -13,7 +13,7 @@
repmgr user permissions
&repmgr; will create an extension in the BDR database containing objects
- for administering &repmgr; metadata. The user defined in the `conninfo`
+ for administering &repmgr; metadata. The user defined in the conninfo
setting must be able to access all objects. Additionally, superuser permissions
are required to install the &repmgr; extension. The easiest way to do this
is create the &repmgr; user as a superuser, however if this is not
diff --git a/doc/repmgrd-bdr.sgml b/doc/repmgrd-bdr.sgml
index c5e32a27..a9227fa2 100644
--- a/doc/repmgrd-bdr.sgml
+++ b/doc/repmgrd-bdr.sgml
@@ -167,5 +167,232 @@
chronological order).
+
+
+ Defining the "event_notification_command"
+
+ Key to "failover" execution is the event_notification_command,
+ which is a user-definable script specified in repmpgr.conf
+ and which should reconfigure the proxy server/ connection pooler to point
+ to the other, still-active node.
+
+
+ Each time &repmgr; (or repmgrd) records an event,
+ it can optionally execute the script defined in
+ event_notification_command to take further action;
+ details of the event will be passed as parameters.
+
+
+ Following placeholders are available to the script:
+
+
+
+
+
+
+
+ node ID
+
+
+
+
+
+
+
+
+ event type
+
+
+
+
+
+
+
+
+ success (1 or 0)
+
+
+
+
+
+
+
+ timestamp
+
+
+
+
+
+
+
+
+ details
+
+
+
+
+
+
+ Note that %c and %a will only be provided during
+ bdr_failover events, which is what is of interest here.
+
+
+ The provided sample script (`scripts/bdr-pgbouncer.sh`) is configured like
+ this:
+
+ event_notification_command='/path/to/bdr-pgbouncer.sh %n %e %s "%c" "%a"'
+
+
+ and parses the configures parameters like this:
+
+ NODE_ID=$1
+ EVENT_TYPE=$2
+ SUCCESS=$3
+ NEXT_CONNINFO=$4
+ NEXT_NODE_NAME=$5
+
+
+ The script also contains some hard-coded values about the PgBouncer
+ configuration for both nodes; these will need to be adjusted for your local environment
+ (ideally the scripts would be maintained as templates and generated by some
+ kind of provisioning system).
+
+
+
+ The script performs following steps:
+
+
+ pauses PgBouncer on all nodes
+
+
+ recreates the PgBouncer configuration file on each
+ node using the information provided by repmgrd
+ (primarily the conninfo string) to configure
+ PgBouncer
+
+
+ reloads the PgBouncer configuration
+
+
+ executes the RESUME command (in PgBouncer)
+
+
+
+
+ Following successful script execution, any connections to PgBouncer on the failed BDR node
+ will be redirected to the active node.
+
+
+
+
+ Node monitoring and failover
+
+ At the intervals specified by monitor_interval_secs
+ in repmgr.conf, repmgrd
+ will ping each node to check if it's available. If a node isn't available,
+ repmgrd will enter failover mode and check reconnect_attempts
+ times at intervals of reconnect_interval to confirm the node is definitely unreachable.
+ This buffer period is necessary to avoid false positives caused by transient
+ network outages.
+
+
+ If the node is still unavailable, repmgrd will enter failover mode and execute
+ the script defined in event_notification_command; an entry will be logged
+ in the repmgr.events table and repmgrd will
+ (unless otherwise configured) resume monitoring of the node in "degraded" mode until it reappears.
+
+
+ repmgrd logfile output during a failover event will look something like this
+ on one node (usually the node which has failed, here node2):
+
+ ...
+ [2017-07-27 21:08:39] [INFO] starting continuous BDR node monitoring
+ [2017-07-27 21:08:39] [INFO] monitoring BDR replication status on node "node2" (ID: 2)
+ [2017-07-27 21:08:55] [INFO] monitoring BDR replication status on node "node2" (ID: 2)
+ [2017-07-27 21:09:11] [INFO] monitoring BDR replication status on node "node2" (ID: 2)
+ [2017-07-27 21:09:23] [WARNING] unable to connect to node node2 (ID 2)
+ [2017-07-27 21:09:23] [INFO] checking state of node 2, 0 of 5 attempts
+ [2017-07-27 21:09:23] [INFO] sleeping 1 seconds until next reconnection attempt
+ [2017-07-27 21:09:24] [INFO] checking state of node 2, 1 of 5 attempts
+ [2017-07-27 21:09:24] [INFO] sleeping 1 seconds until next reconnection attempt
+ [2017-07-27 21:09:25] [INFO] checking state of node 2, 2 of 5 attempts
+ [2017-07-27 21:09:25] [INFO] sleeping 1 seconds until next reconnection attempt
+ [2017-07-27 21:09:26] [INFO] checking state of node 2, 3 of 5 attempts
+ [2017-07-27 21:09:26] [INFO] sleeping 1 seconds until next reconnection attempt
+ [2017-07-27 21:09:27] [INFO] checking state of node 2, 4 of 5 attempts
+ [2017-07-27 21:09:27] [INFO] sleeping 1 seconds until next reconnection attempt
+ [2017-07-27 21:09:28] [WARNING] unable to reconnect to node 2 after 5 attempts
+ [2017-07-27 21:09:28] [NOTICE] setting node record for node 2 to inactive
+ [2017-07-27 21:09:28] [INFO] executing notification command for event "bdr_failover"
+ [2017-07-27 21:09:28] [DETAIL] command is:
+ /path/to/bdr-pgbouncer.sh 2 bdr_failover 1 "host=host=node1 dbname=bdrtest user=repmgr connect_timeout=2" "node1"
+ [2017-07-27 21:09:28] [INFO] node 'node2' (ID: 2) detected as failed; next available node is 'node1' (ID: 1)
+ [2017-07-27 21:09:28] [INFO] monitoring BDR replication status on node "node2" (ID: 2)
+ [2017-07-27 21:09:28] [DETAIL] monitoring node "node2" (ID: 2) in degraded mode
+ ...
+
+
+ Output on the other node (node1) during the same event will look like this:
+
+ ...
+ [2017-07-27 21:08:35] [INFO] starting continuous BDR node monitoring
+ [2017-07-27 21:08:35] [INFO] monitoring BDR replication status on node "node1" (ID: 1)
+ [2017-07-27 21:08:51] [INFO] monitoring BDR replication status on node "node1" (ID: 1)
+ [2017-07-27 21:09:07] [INFO] monitoring BDR replication status on node "node1" (ID: 1)
+ [2017-07-27 21:09:23] [WARNING] unable to connect to node node2 (ID 2)
+ [2017-07-27 21:09:23] [INFO] checking state of node 2, 0 of 5 attempts
+ [2017-07-27 21:09:23] [INFO] sleeping 1 seconds until next reconnection attempt
+ [2017-07-27 21:09:24] [INFO] checking state of node 2, 1 of 5 attempts
+ [2017-07-27 21:09:24] [INFO] sleeping 1 seconds until next reconnection attempt
+ [2017-07-27 21:09:25] [INFO] checking state of node 2, 2 of 5 attempts
+ [2017-07-27 21:09:25] [INFO] sleeping 1 seconds until next reconnection attempt
+ [2017-07-27 21:09:26] [INFO] checking state of node 2, 3 of 5 attempts
+ [2017-07-27 21:09:26] [INFO] sleeping 1 seconds until next reconnection attempt
+ [2017-07-27 21:09:27] [INFO] checking state of node 2, 4 of 5 attempts
+ [2017-07-27 21:09:27] [INFO] sleeping 1 seconds until next reconnection attempt
+ [2017-07-27 21:09:28] [WARNING] unable to reconnect to node 2 after 5 attempts
+ [2017-07-27 21:09:28] [NOTICE] other node's repmgrd is handling failover
+ [2017-07-27 21:09:28] [INFO] monitoring BDR replication status on node "node1" (ID: 1)
+ [2017-07-27 21:09:28] [DETAIL] monitoring node "node2" (ID: 2) in degraded mode
+ ...
+
+
+ This assumes only the PostgreSQL instance on node2 has failed. In this case the
+ repmgrd instance running on node2 has performed the failover. However if
+ the entire server becomes unavailable, repmgrd on node1 will perform
+ the failover.
+
+
+
+ Node recovery
+
+ Following failure of a BDR node, if the node subsequently becomes available again,
+ a bdr_recovery event will be generated. This could potentially be used to
+ reconfigure PgBouncer automatically to bring the node back into the available pool,
+ however it would be prudent to manually verify the node's status before
+ exposing it to the application.
+
+
+ If the failed node comes back up and connects correctly, output similar to this
+ will be visible in the repmgrd log:
+
+ [2017-07-27 21:25:30] [DETAIL] monitoring node "node2" (ID: 2) in degraded mode
+ [2017-07-27 21:25:46] [INFO] monitoring BDR replication status on node "node2" (ID: 2)
+ [2017-07-27 21:25:46] [DETAIL] monitoring node "node2" (ID: 2) in degraded mode
+ [2017-07-27 21:25:55] [INFO] active replication slot for node "node1" found after 1 seconds
+ [2017-07-27 21:25:55] [NOTICE] node "node2" (ID: 2) has recovered after 986 seconds
+
+
+
+
+ Shutdown of both nodes
+
+ If both PostgreSQL instances are shut down, repmgrd will try and handle the
+ situation as gracefully as possible, though with no failover candidates available
+ there's not much it can do. Should this case ever occur, we recommend shutting
+ down repmgrd on both nodes and restarting it once the PostgreSQL instances
+ are running properly.
+
+