From aeee11d1b728a263219967ef9a5331471b009b71 Mon Sep 17 00:00:00 2001 From: Ian Barwick Date: Thu, 26 Oct 2017 18:52:35 +0900 Subject: [PATCH] docs: finalize conversion of existing BDR repmgr documentation --- doc/configuration.sgml | 2 +- doc/repmgrd-bdr.sgml | 227 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 228 insertions(+), 1 deletion(-) diff --git a/doc/configuration.sgml b/doc/configuration.sgml index ffd4e53e..b9cbb68b 100644 --- a/doc/configuration.sgml +++ b/doc/configuration.sgml @@ -13,7 +13,7 @@ repmgr user permissions &repmgr; will create an extension in the BDR database containing objects - for administering &repmgr; metadata. The user defined in the `conninfo` + for administering &repmgr; metadata. The user defined in the conninfo setting must be able to access all objects. Additionally, superuser permissions are required to install the &repmgr; extension. The easiest way to do this is create the &repmgr; user as a superuser, however if this is not diff --git a/doc/repmgrd-bdr.sgml b/doc/repmgrd-bdr.sgml index c5e32a27..a9227fa2 100644 --- a/doc/repmgrd-bdr.sgml +++ b/doc/repmgrd-bdr.sgml @@ -167,5 +167,232 @@ chronological order). + + + Defining the "event_notification_command" + + Key to "failover" execution is the event_notification_command, + which is a user-definable script specified in repmpgr.conf + and which should reconfigure the proxy server/ connection pooler to point + to the other, still-active node. + + + Each time &repmgr; (or repmgrd) records an event, + it can optionally execute the script defined in + event_notification_command to take further action; + details of the event will be passed as parameters. + + + Following placeholders are available to the script: + + + + + + + + node ID + + + + + + + + + event type + + + + + + + + + success (1 or 0) + + + + + + + + timestamp + + + + + + + + + details + + + + + + + Note that %c and %a will only be provided during + bdr_failover events, which is what is of interest here. + + + The provided sample script (`scripts/bdr-pgbouncer.sh`) is configured like + this: + + event_notification_command='/path/to/bdr-pgbouncer.sh %n %e %s "%c" "%a"' + + + and parses the configures parameters like this: + + NODE_ID=$1 + EVENT_TYPE=$2 + SUCCESS=$3 + NEXT_CONNINFO=$4 + NEXT_NODE_NAME=$5 + + + The script also contains some hard-coded values about the PgBouncer + configuration for both nodes; these will need to be adjusted for your local environment + (ideally the scripts would be maintained as templates and generated by some + kind of provisioning system). + + + + The script performs following steps: + + + pauses PgBouncer on all nodes + + + recreates the PgBouncer configuration file on each + node using the information provided by repmgrd + (primarily the conninfo string) to configure + PgBouncer + + + reloads the PgBouncer configuration + + + executes the RESUME command (in PgBouncer) + + + + + Following successful script execution, any connections to PgBouncer on the failed BDR node + will be redirected to the active node. + + + + + Node monitoring and failover + + At the intervals specified by monitor_interval_secs + in repmgr.conf, repmgrd + will ping each node to check if it's available. If a node isn't available, + repmgrd will enter failover mode and check reconnect_attempts + times at intervals of reconnect_interval to confirm the node is definitely unreachable. + This buffer period is necessary to avoid false positives caused by transient + network outages. + + + If the node is still unavailable, repmgrd will enter failover mode and execute + the script defined in event_notification_command; an entry will be logged + in the repmgr.events table and repmgrd will + (unless otherwise configured) resume monitoring of the node in "degraded" mode until it reappears. + + + repmgrd logfile output during a failover event will look something like this + on one node (usually the node which has failed, here node2): + + ... + [2017-07-27 21:08:39] [INFO] starting continuous BDR node monitoring + [2017-07-27 21:08:39] [INFO] monitoring BDR replication status on node "node2" (ID: 2) + [2017-07-27 21:08:55] [INFO] monitoring BDR replication status on node "node2" (ID: 2) + [2017-07-27 21:09:11] [INFO] monitoring BDR replication status on node "node2" (ID: 2) + [2017-07-27 21:09:23] [WARNING] unable to connect to node node2 (ID 2) + [2017-07-27 21:09:23] [INFO] checking state of node 2, 0 of 5 attempts + [2017-07-27 21:09:23] [INFO] sleeping 1 seconds until next reconnection attempt + [2017-07-27 21:09:24] [INFO] checking state of node 2, 1 of 5 attempts + [2017-07-27 21:09:24] [INFO] sleeping 1 seconds until next reconnection attempt + [2017-07-27 21:09:25] [INFO] checking state of node 2, 2 of 5 attempts + [2017-07-27 21:09:25] [INFO] sleeping 1 seconds until next reconnection attempt + [2017-07-27 21:09:26] [INFO] checking state of node 2, 3 of 5 attempts + [2017-07-27 21:09:26] [INFO] sleeping 1 seconds until next reconnection attempt + [2017-07-27 21:09:27] [INFO] checking state of node 2, 4 of 5 attempts + [2017-07-27 21:09:27] [INFO] sleeping 1 seconds until next reconnection attempt + [2017-07-27 21:09:28] [WARNING] unable to reconnect to node 2 after 5 attempts + [2017-07-27 21:09:28] [NOTICE] setting node record for node 2 to inactive + [2017-07-27 21:09:28] [INFO] executing notification command for event "bdr_failover" + [2017-07-27 21:09:28] [DETAIL] command is: + /path/to/bdr-pgbouncer.sh 2 bdr_failover 1 "host=host=node1 dbname=bdrtest user=repmgr connect_timeout=2" "node1" + [2017-07-27 21:09:28] [INFO] node 'node2' (ID: 2) detected as failed; next available node is 'node1' (ID: 1) + [2017-07-27 21:09:28] [INFO] monitoring BDR replication status on node "node2" (ID: 2) + [2017-07-27 21:09:28] [DETAIL] monitoring node "node2" (ID: 2) in degraded mode + ... + + + Output on the other node (node1) during the same event will look like this: + + ... + [2017-07-27 21:08:35] [INFO] starting continuous BDR node monitoring + [2017-07-27 21:08:35] [INFO] monitoring BDR replication status on node "node1" (ID: 1) + [2017-07-27 21:08:51] [INFO] monitoring BDR replication status on node "node1" (ID: 1) + [2017-07-27 21:09:07] [INFO] monitoring BDR replication status on node "node1" (ID: 1) + [2017-07-27 21:09:23] [WARNING] unable to connect to node node2 (ID 2) + [2017-07-27 21:09:23] [INFO] checking state of node 2, 0 of 5 attempts + [2017-07-27 21:09:23] [INFO] sleeping 1 seconds until next reconnection attempt + [2017-07-27 21:09:24] [INFO] checking state of node 2, 1 of 5 attempts + [2017-07-27 21:09:24] [INFO] sleeping 1 seconds until next reconnection attempt + [2017-07-27 21:09:25] [INFO] checking state of node 2, 2 of 5 attempts + [2017-07-27 21:09:25] [INFO] sleeping 1 seconds until next reconnection attempt + [2017-07-27 21:09:26] [INFO] checking state of node 2, 3 of 5 attempts + [2017-07-27 21:09:26] [INFO] sleeping 1 seconds until next reconnection attempt + [2017-07-27 21:09:27] [INFO] checking state of node 2, 4 of 5 attempts + [2017-07-27 21:09:27] [INFO] sleeping 1 seconds until next reconnection attempt + [2017-07-27 21:09:28] [WARNING] unable to reconnect to node 2 after 5 attempts + [2017-07-27 21:09:28] [NOTICE] other node's repmgrd is handling failover + [2017-07-27 21:09:28] [INFO] monitoring BDR replication status on node "node1" (ID: 1) + [2017-07-27 21:09:28] [DETAIL] monitoring node "node2" (ID: 2) in degraded mode + ... + + + This assumes only the PostgreSQL instance on node2 has failed. In this case the + repmgrd instance running on node2 has performed the failover. However if + the entire server becomes unavailable, repmgrd on node1 will perform + the failover. + + + + Node recovery + + Following failure of a BDR node, if the node subsequently becomes available again, + a bdr_recovery event will be generated. This could potentially be used to + reconfigure PgBouncer automatically to bring the node back into the available pool, + however it would be prudent to manually verify the node's status before + exposing it to the application. + + + If the failed node comes back up and connects correctly, output similar to this + will be visible in the repmgrd log: + + [2017-07-27 21:25:30] [DETAIL] monitoring node "node2" (ID: 2) in degraded mode + [2017-07-27 21:25:46] [INFO] monitoring BDR replication status on node "node2" (ID: 2) + [2017-07-27 21:25:46] [DETAIL] monitoring node "node2" (ID: 2) in degraded mode + [2017-07-27 21:25:55] [INFO] active replication slot for node "node1" found after 1 seconds + [2017-07-27 21:25:55] [NOTICE] node "node2" (ID: 2) has recovered after 986 seconds + + + + + Shutdown of both nodes + + If both PostgreSQL instances are shut down, repmgrd will try and handle the + situation as gracefully as possible, though with no failover candidates available + there's not much it can do. Should this case ever occur, we recommend shutting + down repmgrd on both nodes and restarting it once the PostgreSQL instances + are running properly. + +