Overview of repmgr commands. repmgr standby clone cloning repmgr standby clone clones a PostgreSQL node from another PostgreSQL node, typically the primary, but optionally from any other node in the cluster or from Barman. It creates the recovery.conf file required to attach the cloned node to the primary node (or another standby, if cascading replication is in use). repmgr standby clone does not start the standby, and after cloning repmgr standby register must be executed to notify &repmgr; of its presence. Note that by default, all configuration files in the source node's data directory will be copied to the cloned node. Typically these will be postgresql.conf, postgresql.auto.conf, pg_hba.conf and pg_ident.conf. These may require modification before the standby is started. In some cases (e.g. on Debian or Ubuntu Linux installations), PostgreSQL's configuration files are located outside of the data directory and will not be copied by default. &repmgr; can copy these files, either to the same location on the standby server (provided appropriate directory and file permissions are available), or into the standby's data directory. This requires passwordless SSH access to the primary server. Add the option --copy-external-config-files to the repmgr standby clone command; by default files will be copied to the same path as on the upstream server. Note that the user executing repmgr must have write access to those directories. To have the configuration files placed in the standby's data directory, specify --copy-external-config-files=pgdata, but note that any include directives in the copied files may need to be updated. For reliable configuration file management we recommend using a configuration management tool such as Ansible, Chef, Puppet or Salt. When initially cloning a standby, you will need to ensure that all required WAL files remain available while the cloning is taking place. To ensure this happens when using the default `pg_basebackup` method, &repmgr; will set pg_basebackup's --xlog-method parameter to stream, which will ensure all WAL files generated during the cloning process are streamed in parallel with the main backup. Note that this requires two replication connections to be available (&repmgr; will verify sufficient connections are available before attempting to clone, and this can be checked before performing the clone using the --dry-run option). To override this behaviour, in repmgr.conf set pg_basebackup's --xlog-method parameter to fetch: pg_basebackup_options='--xlog-method=fetch' and ensure that wal_keep_segments is set to an appropriately high value. See the pg_basebackup documentation for details. From PostgreSQL 10, pg_basebackup's --xlog-method parameter has been renamed to --wal-method. repmgr standby register repmgr standby register adds a standby's information to the &repmgr; metadata. This command needs to be executed to enable promote/follow operations and to allow repmgrd to work with the node. An existing standby can be registered using this command. Execute with the --dry-run option to check what would happen without actually registering the standby. Depending on your environment and workload, it may take some time for the standby's node record to propagate from the primary to the standby. Some actions (such as starting repmgrd) require that the standby's node record is present and up-to-date to function correctly. By providing the option --wait-sync to the repmgr standby register command, &repmgr; will wait until the record is synchronised before exiting. An optional timeout (in seconds) can be added to this option (e.g. --wait-sync=60). Under some circumstances you may wish to register a standby which is not yet running; this can be the case when using provisioning tools to create a complex replication cluster. In this case, by using the -F/--force option and providing the connection parameters to the primary server, the standby can be registered. Similarly, with cascading replication it may be necessary to register a standby whose upstream node has not yet been registered - in this case, using -F/--force will result in the creation of an inactive placeholder record for the upstream node, which will however later need to be registered with the -F/--force option too. When used with repmgr standby register, care should be taken that use of the -F/--force option does not result in an incorrectly configured cluster. repmgr standby promote Promotes a standby to a primary if the current primary has failed. This command requires a valid repmgr.conf file for the standby, either specified explicitly with -f/--config-file or located in a default location; no additional arguments are required. If the standby promotion succeeds, the server will not need to be restarted. However any other standbys will need to follow the new server, by using ; if repmgrd is active, it will handle this automatically. repmgr standby follow Attaches the standby to a new primary. This command requires a valid repmgr.conf file for the standby, either specified explicitly with -f/--config-file or located in a default location; no additional arguments are required. This command will force a restart of the standby server, which must be running. It can only be used to attach a standby to a new primary node. To re-add an inactive node to the replication cluster, see repmgr node rejoin Enables a dormant (stopped) node to be rejoined to the replication cluster. This can optionally use pg_rewind to re-integrate a node which has diverged from the rest of the cluster, typically a failed primary. repmgr cluster show Displays information about each active node in the replication cluster. This command polls each registered server and shows its role (primary / standby / bdr) and status. It polls each server directly and can be run on any node in the cluster; this is also useful when analyzing connectivity from a particular node. This command requires either a valid repmgr.conf file or a database connection string to one of the registered nodes; no additional arguments are needed. Example: $ repmgr -f /etc/repmgr.conf cluster show ID | Name | Role | Status | Upstream | Location | Connection string ----+-------+---------+-----------+----------+----------+----------------------------------------- 1 | node1 | primary | * running | | default | host=db_node1 dbname=repmgr user=repmgr 2 | node2 | standby | running | node1 | default | host=db_node2 dbname=repmgr user=repmgr 3 | node3 | standby | running | node1 | default | host=db_node3 dbname=repmgr user=repmgr To show database connection errors when polling nodes, run the command in --verbose mode. The `cluster show` command accepts an optional parameter --csv, which outputs the replication cluster's status in a simple CSV format, suitable for parsing by scripts: $ repmgr -f /etc/repmgr.conf cluster show --csv 1,-1,-1 2,0,0 3,0,1 The columns have following meanings: node ID availability (0 = available, -1 = unavailable) recovery state (0 = not in recovery, 1 = in recovery, -1 = unknown) Note that the availability is tested by connecting from the node where repmgr cluster show is executed, and does not necessarily imply the node is down. See and to get a better overviews of connections between nodes. repmgr cluster matrix repmgr cluster matrix runs repmgr cluster show on each node and arranges the results in a matrix, recording success or failure. repmgr cluster matrix requires a valid repmgr.conf file on each node. Additionally passwordless `ssh` connections are required between all nodes. Example 1 (all nodes up): $ repmgr -f /etc/repmgr.conf cluster matrix Name | Id | 1 | 2 | 3 -------+----+----+----+---- node1 | 1 | * | * | * node2 | 2 | * | * | * node3 | 3 | * | * | * Example 2 (node1 and node2 up, node3 down): $ repmgr -f /etc/repmgr.conf cluster matrix Name | Id | 1 | 2 | 3 -------+----+----+----+---- node1 | 1 | * | * | x node2 | 2 | * | * | x node3 | 3 | ? | ? | ? Each row corresponds to one server, and indicates the result of testing an outbound connection from that server. Since node3 is down, all the entries in its row are filled with ?, meaning that there we cannot test outbound connections. The other two nodes are up; the corresponding rows have x in the column corresponding to node3, meaning that inbound connections to that node have failed, and `*` in the columns corresponding to node1 and node2, meaning that inbound connections to these nodes have succeeded. Example 3 (all nodes up, firewall dropping packets originating from node1 and directed to port 5432 on node3) - running repmgr cluster matrix from node1 gives the following output: $ repmgr -f /etc/repmgr.conf cluster matrix Name | Id | 1 | 2 | 3 -------+----+----+----+---- node1 | 1 | * | * | x node2 | 2 | * | * | * node3 | 3 | ? | ? | ? Note this may take some time depending on the connect_timeout setting in the node conninfo strings; default is 1 minute which means without modification the above command would take around 2 minutes to run; see comment elsewhere about setting connect_timeout) The matrix tells us that we cannot connect from node1 to node3, and that (therefore) we don't know the state of any outbound connection from node3. In this case, the command will produce a more useful result. repmgr cluster crosscheck repmgr cluster crosscheck is similar to , but cross-checks connections between each combination of nodes. In "Example 3" in we have no information about the state of node3. However by running repmgr cluster crosscheck it's possible to get a better overview of the cluster situation: $ repmgr -f /etc/repmgr.conf cluster crosscheck Name | Id | 1 | 2 | 3 -------+----+----+----+---- node1 | 1 | * | * | x node2 | 2 | * | * | * node3 | 3 | * | * | * What happened is that repmgr cluster crosscheck merged its own repmgr cluster matrix with the repmgr cluster matrix output from node2; the latter is able to connect to node3 and therefore determine the state ofx outbound connections from that node.