repmgr: Replication Manager for PostgreSQL ========================================== `repmgr` is a suite of open-source tools to manage replication and failover within a cluster of PostgreSQL servers. It enhances PostgreSQL's built-in replication capabilities with utilities to set up standby servers, monitor replication, and perform administrative tasks such as failover or switchover operations. `repmgr 4` is a complete rewrite of the existing `repmgr` codebase, allowing the use of all of the latest features in PostgreSQL replication. PostgreSQL 10, 9.6 and 9.5 are fully supported. PostgreSQL 9.4 and 9.3 are supported, with some restrictions. ### BDR support `repmgr 4` supports monitoring of a two-node BDR 2.0 cluster. PostgreSQL 9.6 is required for BDR 2.0. Note that BDR 2.0 is not publicly available; please contact 2ndQuadrant for details. `repmgr 4` will support future public BDR releases. Changes in repmgr4 and backwards compatibility ----------------------------------------------- `repmgr` is now implemented as a PostgreSQL extension, and all database objects used by repmgr are stored in a dedicated `repmgr` schema, rather than `repmgr_$cluster_name`. Note there is no need to install the extension, this will be done automatically by `repmgr primary register`. Some configuration items have had their names changed for consistency and clarity, e.g. `node` => `node_id`. `repmgr` will issue a warning about deprecated/altered options. Some configuration items have been changed to command line options, and vice- versa, e.g. to avoid hard-coding items such as a a node's upstream ID, which might change over time. See file `doc/changes-in-repmgr4.md` for more details. To upgrade from repmgr 3.x, both the `repmgr` metadatabase and all repmgr configuration files need to be converted. This is quite straightforward and scripts are provided to assist with this. See document `doc/upgrading-from-repmgr3.md` for further details. Overview -------- The `repmgr` suite provides two main tools: - `repmgr` - a command-line tool used to perform administrative tasks such as: - setting up standby servers - promoting a standby server to primary - switching over primary and standby servers - displaying the status of servers in the replication cluster - `repmgrd` is a daemon which actively monitors servers in a replication cluster and performs the following tasks: - monitoring and recording replication performance - performing failover by detecting failure of the primary and promoting the most suitable standby server - provide notifications about events in the cluster to a user-defined script which can perform tasks such as sending alerts by email `repmgr` supports and enhances PostgreSQL's built-in streaming replication, which provides a single read/write primary server and one or more read-only standbys containing near-real time copies of the primary server's database. ### Concepts This guide assumes that you are familiar with PostgreSQL administration and streaming replication concepts. For further details on streaming replication, see this link: https://www.postgresql.org/docs/current/interactive/warm-standby.html#STREAMING-REPLICATION The following terms are used throughout the `repmgr` documentation. - `replication cluster` In the `repmgr` documentation, "replication cluster" refers to the network of PostgreSQL servers connected by streaming replication. - `node` A `node` is a server within a replication cluster. - `upstream node` This is the node a standby server is connected to; either the primary server or in the case of cascading replication, another standby. - `failover` This is the action which occurs if a primary server fails and a suitable standby is promoted as the new primary. The `repmgrd` daemon supports automatic failover to minimise downtime. - `switchover` In certain circumstances, such as hardware or operating system maintenance, it's necessary to take a primary server offline; in this case a controlled switchover is necessary, whereby a suitable standby is promoted and the existing primary removed from the replication cluster in a controlled manner. The `repmgr` command line client provides this functionality. ### repmgr user and metadata In order to effectively manage a replication cluster, `repmgr` needs to store information about the servers in the cluster in a dedicated database schema. This schema is automatically by the `repmgr` extension, which is installed during the first step in initialising a `repmgr`-administered cluster (`repmgr primary register`) and contains the following objects: tables: - `repmgr.events`: records events of interest - `repmgr.nodes`: connection and status information for each server in the replication cluster - `repmgr.monitoring_history`: historical standby monitoring information written by `repmgrd` views: - `repmgr.show_nodes`: based on the table `repl_nodes`, additionally showing the name of the server's upstream node - `repmgr.replication_status`: when `repmgrd`'s monitoring is enabled, shows current monitoring status for each standby. The `repmgr` metadata schema can be stored in an existing database or in its own dedicated database. Note that the `repmgr` metadata schema cannot reside on a database server which is not part of the replication cluster managed by `repmgr`. A database user must be available for `repmgr` to access this database and perform necessary changes. This user does not need to be a superuser, however some operations such as initial installation of the `repmgr` extension will require a superuser connection (this can be specified where required with the command line option `--superuser`). Installation ------------ ### System requirements `repmgr` is developed and tested on Linux and OS X, but should work on any UNIX-like system supported by PostgreSQL itself. `repmgr 4` supports PostgreSQL from version 9.3. If upgrading from `repmgr 3`, please see the separate upgrade guide `doc/upgrading-from-repmgr3.md`. All servers in the replication cluster must be running the same major version of PostgreSQL, and we recommend that they also run the same minor version. The `repmgr` tools must be installed on each server in the replication cluster. A dedicated system user for `repmgr` is *not* required; as many `repmgr` and `repmgrd` actions require direct access to the PostgreSQL data directory, these commands should be executed by the `postgres` user. Passwordless `ssh` connectivity between all servers in the replication cluster is not required, but is necessary in the following cases: * if you need `repmgr` to copy configuration files from outside the PostgreSQL data directory (in which case `rsync` is also required) * to perform switchover operations * when executing `repmgr cluster matrix` and `repmgr cluster crosscheck` * * * > *TIP*: We recommend using a session multiplexer utility such as `screen` or > `tmux` when performing long-running actions (such as cloning a database) > on a remote server - this will ensure the `repmgr` action won't be prematurely > terminated if your `ssh` session to the server is interrupted or closed. * * * ### Packages We recommend installing `repmgr` using the available packages for your system. - RedHat/CentOS: RPM packages for `repmgr` are available via Yum through the PostgreSQL Global Development Group RPM repository ( http://yum.postgresql.org/ ). Follow the instructions for your distribution (RedHat, CentOS, Fedora, etc.) and architecture as detailed at yum.postgresql.org. 2ndQuadrant also provides its own RPM packages which are made available at the same time as each `repmgr` release, as it can take some days for them to become available via the main PGDG repository. See here for details: http://repmgr.org/yum-repository.html - Debian/Ubuntu: the most recent `repmgr` packages are available from the PostgreSQL Community APT repository ( http://apt.postgresql.org/ ). Instructions can be found in the APT section of the PostgreSQL Wiki ( https://wiki.postgresql.org/wiki/Apt ). See `PACKAGES.md` for details on building .deb and .rpm packages from the `repmgr` source code. ### Source installation `repmgr` source code can be obtained directly from the project GitHub repository: git clone https://github.com/2ndQuadrant/repmgr Release tarballs are also available: https://github.com/2ndQuadrant/repmgr/releases http://repmgr.org/ ### Building from source Simply: ./configure && make install Ensure `pg_config` for the target PostgreSQL version is in `$PATH`. Configuration ------------- `repmgr` and `repmgrd` use a common configuration file, by default called `repmgr.conf` (although any name can be used if explicitly specified). `repmgr.conf` must contain a number of required parameters, including the database connection string for the local node and the location of its data directory; other values will be inferred from defaults if not explicitly supplied. See section `repmgr configuration file` below for more details. The configuration file will be searched for in the following locations: - a configuration file specified by the `-f/--config-file` command line option - a location specified by the package maintainer (if `repmgr` was installed from a package and the package maintainer has specified the configuration file location) - `repmgr.conf` in the local directory - `/etc/repmgr.conf` - the directory reported by `pg_config --sysconfdir` Note that if a file is explicitly specified with `-f/--config-file`, an error will be raised if it is not found or not readable and no attempt will be made to check default locations; this is to prevent `repmgr` unexpectedly reading the wrong file. For a full list of annotated configuration items, see the file `repmgr.conf.sample`. The following parameters in the configuration file can be overridden with command line options: - `log_level` with `-L/--log-level` - `pg_bindir` with `-b/--pg_bindir` ### Logging By default `repmgr` and `repmgrd` will log directly to `STDERR`. For `repmgrd` we recommend capturing output in a logfile or using your system's log facility; see `repmgr.conf.sample` for details. As a command line utility, `repmgr` will log directly to the console by default. However in some circumstances, such as when `repmgr` is executed by `repmgrd` during a failover event, it makes sense to capture `repmgr`'s log output - this can be done by supplying the command-line option `--log-to-file` to `repmgr`. ### Command line options and environment variables For some commands, e.g. `repmgr standby clone`, database connection parameters need to be provided. Like other PostgreSQL utilities, following standard parameters can be used: - `-d/--dbname=DBNAME` - `-h/--host=HOSTNAME` - `-p/--port=PORT` - `-U/--username=USERNAME` If `-d/--dbname` contains an `=` sign or starts with a valid URI prefix (`postgresql://` or `postgres://`), it is treated as a conninfo string. See the PostgreSQL documentation for further details: https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-CONNSTRING Note that if a `conninfo` string is provided, values set in this will override any provided as individual parameters. For example, with `-d 'host=foo' --host bar`, `foo` will be chosen over `bar`. Like other PostgreSQL utilities based on `libpq`, `repmgr` will default to any values set in environment variables if explicit command line parameters are not provided. See the PostgreSQL documentation for further details: https://www.postgresql.org/docs/current/static/libpq-envars.html Setting up a basic replication cluster with repmgr --------------------------------------------------- The following section will describe how to set up a basic replication cluster with a primary and a standby server using the `repmgr` command line tool. It is assumed PostgreSQL is installed on both servers in the cluster, `rsync` is available and passwordless SSH connections are possible between both servers. * * * > *TIP*: for testing `repmgr`, it's possible to use multiple PostgreSQL > instances running on different ports on the same computer, with > passwordless SSH access to `localhost` enabled. * * * ### PostgreSQL configuration On the primary server, a PostgreSQL instance must be initialised and running. The following replication settings may need to be adjusted: # Enable replication connections; set this figure to at least one more # than the number of standbys which will connect to this server # (note that repmgr will execute `pg_basebackup` in WAL streaming mode, # which requires two free WAL senders) max_wal_senders = 10 # Ensure WAL files contain enough information to enable read-only queries # on the standby. # # PostgreSQL 9.5 and earlier: one of 'hot_standby' or 'logical' # PostgreSQL 9.6 and later: one of 'replica' or 'logical' # ('hot_standby' will still be accepted as an alias for 'replica') # # See: https://www.postgresql.org/docs/current/static/runtime-config-wal.html#GUC-WAL-LEVEL wal_level = 'hot_standby' # Enable read-only queries on a standby # (Note: this will be ignored on a primary but we recommend including # it anyway) hot_standby = on # Enable WAL file archiving archive_mode = on # Set archive command to a script or application that will safely store # you WALs in a secure place. /bin/true is an example of a command that # ignores archiving. Use something more sensible. archive_command = '/bin/true' # If you have configured `pg_basebackup_options` # in `repmgr.conf` to include the setting `--xlog-method=fetch` (from # PostgreSQL 10 `--wal-method=fetch`), *and* you have not set # `restore_command` in `repmgr.conf`to fetch WAL files from another # source such as Barman, you'll need to set `wal_keep_segments` to a # high enough value to ensure that all WAL files generated while # the standby is being cloned are retained until the standby starts up. # # wal_keep_segments = 5000 * * * > *TIP*: rather than editing these settings in the default `postgresql.conf` > file, create a separate file such as `postgresql.replication.conf` and > include it from the end of the main configuration file with: > `include 'postgresql.replication.conf'` * * * Create a dedicated PostgreSQL superuser account and a database for the `repmgr` metadata, e.g. createuser -s repmgr createdb repmgr -O repmgr For the examples in this document, the name `repmgr` will be used for both user and database, but any names can be used. Ensure the `repmgr` user has appropriate permissions in `pg_hba.conf` and can connect in replication mode; `pg_hba.conf` should contain entries similar to the following: local replication repmgr trust host replication repmgr 127.0.0.1/32 trust host replication repmgr 192.168.1.0/24 trust local repmgr repmgr trust host repmgr repmgr 127.0.0.1/32 trust host repmgr repmgr 192.168.1.0/24 trust Adjust according to your network environment and authentication requirements. On the standby, do not create a PostgreSQL instance, but do ensure an empty directory is available for the `postgres` system user to create a data directory. ### repmgr configuration file Create a `repmgr.conf` file on the primary server. The file must contain at least the following parameters: node_id=1 node_name=node1 conninfo='host=node1 user=repmgr dbname=repmgr connect_timeout=2' data_directory='/var/lib/postgresql/data' - `node_id`: a unique integer identifying the node; note this must be a positive 32 bit signed integer between 1 and 2147483647 - `node_name`: a unique string identifying the node; we recommend a name specific to the server (e.g. 'server_1'); avoid names indicating the current replication role like 'primary' or 'standby' as the server's role could change. - `conninfo`: a valid connection string for the `repmgr` database on the *current* server. (On the standby, the database will not yet exist, but `repmgr` needs to know the connection details to complete the setup process). - `data_directory`: the node's data directory `repmgr.conf` should not be stored inside the PostgreSQL data directory, as it could be overwritten when setting up or reinitialising the PostgreSQL server. See section `Configuration` above for further details about `repmgr.conf`. `repmgr` will install the `repmgr` extension, which creates a `repmgr` schema containing the `repmgr` metadata tables as well as other functions and views. We also recommend that you set the `repmgr` user's search path to include this schema name, e.g. ALTER USER repmgr SET search_path TO repmgr, "$user", public; * * * > *TIP*: for Debian-based distributions we recommend explicitly setting > `pg_bindir` to the directory where `pg_ctl` and other binaries not in > the standard path are located. For PostgreSQL 9.6 this would be > `/usr/lib/postgresql/9.6/bin/`. * * * See `repmgr.conf.sample` for details of all available configuration parameters. ### Register the primary server To enable `repmgr` to support a replication cluster, the primary node must be registered with `repmgr`. This installs the `repmgr` extension and metadata objects, and adds a metadata record for the primary server: $ repmgr -f repmgr.conf primary register INFO: connecting to primary database... NOTICE: attempting to install extension "repmgr" NOTICE: "repmgr" extension successfully installed NOTICE: primary node record (id: 1) registered Verify status of the cluster like this: ID | Name | Role | Status | Upstream | Connection string ----+-------+---------+-----------+----------+-------------------------------------- 1 | node1 | primary | * running | | host=node1 dbname=repmgr user=repmgr The record in the `repmgr` metadata table will look like this: repmgr=# SELECT * from repmgr.nodes; -[ RECORD 1 ]----+--------------------------------------- node_id | 1 upstream_node_id | active | t node_name | node1 type | primary location | default priority | 100 conninfo | host=node1 dbname=repmgr user=repmgr repluser | repmgr slot_name | config_file | /etc/repmgr.conf Each server in the replication cluster will have its own record and will be updated when its status or role changes. ### Clone the standby server Create a `repmgr.conf` file on the standby server. It must contain at least the same parameters as the primary's `repmgr.conf`, but with the mandatory values `node`, `node_name`, `conninfo` (and possibly `data_directory`) adjusted accordingly, e.g.: node=2 node_name=node2 conninfo='host=node2 user=repmgr dbname=repmgr' data_directory='/var/lib/postgresql/data' Clone the standby with: $ repmgr -h node1 -U repmgr -d repmgr -D /path/to/node2/data/ -f /etc/repmgr.conf standby clone NOTICE: using configuration file "/etc/repmgr.conf" NOTICE: destination directory "/var/lib/postgresql/data" provided INFO: connecting to upstream node INFO: connected to source node, checking its state NOTICE: checking for available walsenders on upstream node (2 required) INFO: sufficient walsenders available on upstream node (2 required) INFO: successfully connected to source node DETAIL: current installation size is 29 MB INFO: creating directory "/var/lib/postgresql/data"... NOTICE: starting backup (using pg_basebackup)... HINT: this may take some time; consider using the -c/--fast-checkpoint option INFO: executing: 'pg_basebackup -l "repmgr base backup" -D /var/lib/postgresql/data -h node1 -U repmgr -X stream ' NOTICE: standby clone (using pg_basebackup) complete NOTICE: you can now start your PostgreSQL server HINT: for example: pg_ctl -D /var/lib/postgresql//data start This will clone the PostgreSQL data directory files from the primary at `node1` using PostgreSQL's `pg_basebackup` utility. A `recovery.conf` file containing the correct parameters to start streaming from this primary server will be created automatically. Note that by default, any configuration files in the primary's data directory will be copied to the standby. Typically these will be `postgresql.conf`, `postgresql.auto.conf`, `pg_hba.conf` and `pg_ident.conf`. These may require modification before the standby is started so it functions as desired. In some cases (e.g. on Debian or Ubuntu Linux installations), PostgreSQL's configuration files are located outside of the data directory and will not be copied by default. `repmgr` can copy these files, either to the same location on the standby server (provided appropriate directory and file permissions are available), or into the standby's data directory. This requires passwordless SSH access to the primary server. Add the option `--copy-external-config-files` to the `repmgr standby clone` command; by default files will be copied to the same path as on the upstream server. To have them placed in the standby's data directory, specify `--copy-external-config-files=pgdata`, but note that any include directives in the copied files may need to be updated. *Caveat*: when copying external configuration files: `repmgr` will only be able to detect files which contain active settings. If a file is referenced by an include directive but is empty, only contains comments or contains settings which have not been activated, the file will not be copied. * * * > *TIP*: for reliable configuration file management we recommend using a > configuration management tool such as Ansible, Chef, Puppet or Salt. * * * Be aware that when initially cloning a standby, you will need to ensure that all required WAL files remain available while the cloning is taking place. To ensure this happens when using the default `pg_basebackup` method, `repmgr` will set `pg_basebackup`'s `--xlog-method` parameter to `stream`, which will ensure all WAL files generated during the cloning process are streamed in parallel with the main backup. Note that this requires two replication connections to be available (`repmgr` will verify sufficient connections are available before attempting to clone). To override this behaviour, in `repmgr.conf` set `pg_basebackup`'s `--xlog-method` parameter to `fetch`: pg_basebackup_options='--xlog-method=fetch' and ensure that `wal_keep_segments` is set to an appropriately high value. See the `pg_basebackup` documentation for details: https://www.postgresql.org/docs/current/static/app-pgbasebackup.html > *NOTE*: From PostgreSQL 10, `pg_basebackup`'s `--xlog-method` parameter > has been renamed to `--wal-method`. Make any adjustments to the standby's PostgreSQL configuration files now, then start the server. * * * > *NOTE*: `repmgr standby clone` does not require `repmgr.conf`, however we > recommend providing this as `repmgr` will set the `application_name` parameter > in `recovery.conf` as the value provided in `node_name`, making it easier to > identify the node in `pg_stat_replication`. It's also possible to provide some > advanced options for controlling the standby cloning process; see next section > for details. * * * ### Verify replication is functioning Connect to the primary server and execute: repmgr=# SELECT * FROM pg_stat_replication; -[ RECORD 1 ]----+------------------------------ pid | 19111 usesysid | 16384 usename | repmgr application_name | node2 client_addr | ::1 client_hostname | client_port | 50378 backend_start | 2017-08-28 15:14:19.851581+09 backend_xmin | state | streaming sent_location | 0/7000318 write_location | 0/7000318 flush_location | 0/7000318 replay_location | 0/7000318 sync_priority | 0 sync_state | async ### Register the standby Register the standby server with: $ repmgr -f /etc/repmgr.conf standby register NOTICE: standby node "node2" (id: 2) successfully registered Check the node is registered by executing `repmgr cluster show` on the standby: $ repmgr -f /etc/repmgr.conf cluster show ID | Name | Role | Status | Upstream | Location | Connection string ----+-------+---------+-----------+----------+----------+-------------------------------------- 1 | node1 | primary | * running | | default | host=node1 dbname=repmgr user=repmgr 2 | node2 | standby | running | node1 | default | host=node2 dbname=repmgr user=repmgr The standby server now has a copy of the records for all servers in the replication cluster. * * * > *TIP*: depending on your environment and workload, it may take some time for > the standby's node record to propagate from the primary to the standby. Some > actions (such as starting `repmgrd`) require that the standby's node record > is present and up-to-date to function correctly - by providing the option > `--wait-sync` to the `repmgr standby register` command, `repmgr` will wait > until the record is synchronised before exiting. An optional timeout (in > seconds) can be added to this option (e.g. `--wait-sync=60`). * * * Under some circumstances you may wish to register a standby which is not yet running; this can be the case when using provisioning tools to create a complex replication cluster. In this case, by using the `-F/--force` option and providing the connection parameters to the primary server, the standby can be registered. Similarly, with cascading replication it may be necessary to register a standby whose upstream node has not yet been registered - in this case, using `-F/--force` will result in the creation of an inactive placeholder record for the upstream node, which will however later need to be registered with the `-F/--force` option too. When used with `standby register`, care should be taken that use of the `-F/--force` option does not result in an incorrectly configured cluster. ### Using Barman to clone a standby `repmgr standby clone` can use Barman (the "Backup and Replication manager", https://www.pgbarman.org/), as a provider of both base backups and WAL files. Barman support provides the following advantages: - the primary node does not need to perform a new backup every time a new standby is cloned; - a standby node can be disconnected for longer periods without losing the ability to catch up, and without causing accumulation of WAL files on the primary node; - therefore, `repmgr` does not need to use replication slots, and on the primary node, `wal_keep_segments` does not need to be set. > *NOTE*: In view of the above, Barman support is incompatible with > the `use_replication_slots` setting in `repmgr.conf`. In order to enable Barman support for `repmgr standby clone`, following prerequisites must be met: - the `barman_server` setting in `repmgr.conf` is the same as the server configured in Barman; - the `barman_host` setting in `repmgr.conf` is set to the SSH hostname of the Barman server; - the `restore_command` setting in `repmgr.conf` is configured to use a copy of the `barman-wal-restore` script shipped with the `barman-cli` package (see below); - the Barman catalogue includes at least one valid backup for this server. > *NOTE*: Barman support is automatically enabled if `barman_server` > is set. Normally it is a good practice to use Barman, for instance > when fetching a base backup while cloning a standby; in any case, > Barman mode can be disabled using the `--without-barman` command > line option. > *NOTE*: if you have a non-default SSH configuration on the Barman > server, e.g. using a port other than 22, then you can set those > parameters in a dedicated Host section in `~/.ssh/config` > corresponding to the value of `barman_server` in `repmgr.conf`. See > the "Host" section in `man 5 ssh_config` for more details. `barman-wal-restore` is a Python script provided by the Barman development team as part of the `barman-cli` package (Barman 2.0 and later; for Barman 1.x the script is provided separately as `barman-wal-restore.py`). `restore_command` must then be set in `repmgr.conf` as follows: