mirror of
https://github.com/EnterpriseDB/repmgr.git
synced 2026-03-22 22:56:29 +00:00
8edc64f64e
Sadly it's likely there are still installations running PostgreSQL versions released prior to February 2021, so keep the note there for now.
506 lines
21 KiB
XML
506 lines
21 KiB
XML
<refentry id="repmgr-node-rejoin">
|
|
|
|
<indexterm>
|
|
<primary>repmgr node rejoin</primary>
|
|
</indexterm>
|
|
|
|
<refmeta>
|
|
<refentrytitle>repmgr node rejoin</refentrytitle>
|
|
</refmeta>
|
|
|
|
<refnamediv>
|
|
<refname>repmgr node rejoin</refname>
|
|
<refpurpose>rejoin a dormant (stopped) node to the replication cluster</refpurpose>
|
|
</refnamediv>
|
|
|
|
<refsect1>
|
|
<title>Description</title>
|
|
<para>
|
|
Enables a dormant (stopped) node to be rejoined to the replication cluster.
|
|
</para>
|
|
<para>
|
|
This can optionally use <application>pg_rewind</application> to re-integrate
|
|
a node which has diverged from the rest of the cluster, typically a failed primary.
|
|
</para>
|
|
<para>
|
|
Note that <command>repmgr node rejoin</command> can only be used to attach
|
|
a standby to the current primary, not another standby.
|
|
</para>
|
|
|
|
<tip>
|
|
<para>
|
|
If the node is running and needs to be attached to the current primary, use
|
|
<xref linkend="repmgr-standby-follow"/>.
|
|
</para>
|
|
<para>
|
|
Note <xref linkend="repmgr-standby-follow"/> can only be used for standbys which have not diverged
|
|
from the rest of the cluster.
|
|
</para>
|
|
</tip>
|
|
</refsect1>
|
|
|
|
|
|
<refsect1>
|
|
<title>Usage</title>
|
|
|
|
<para>
|
|
<programlisting>
|
|
repmgr node rejoin -d '$conninfo'</programlisting>
|
|
|
|
where <literal>$conninfo</literal> is the PostgreSQL <literal>conninfo</literal> string of the
|
|
<emphasis>current</emphasis> primary node (or that of any reachable node in the cluster, but
|
|
<emphasis>not</emphasis> the local node). This is so that &repmgr; can fetch up-to-date information
|
|
about the current state of the cluster.
|
|
</para>
|
|
<para>
|
|
<filename>repmgr.conf</filename> for the stopped node *must* be supplied explicitly if not
|
|
otherwise available.
|
|
</para>
|
|
</refsect1>
|
|
|
|
<refsect1>
|
|
|
|
<title>Options</title>
|
|
<variablelist>
|
|
|
|
<varlistentry>
|
|
<term><option>--dry-run</option></term>
|
|
<listitem>
|
|
<para>
|
|
Check prerequisites but don't actually execute the rejoin.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><option>--force-rewind</option></term>
|
|
<listitem>
|
|
<para>
|
|
Execute <application>pg_rewind</application>.
|
|
</para>
|
|
<para>
|
|
See <xref linkend="repmgr-node-rejoin-pg-rewind"/> for more details on using
|
|
<application>pg_rewind</application>.
|
|
</para>
|
|
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><option>--config-files</option></term>
|
|
<listitem>
|
|
<para>
|
|
comma-separated list of configuration files to retain after
|
|
executing <application>pg_rewind</application>.
|
|
</para>
|
|
<para>
|
|
Currently <application>pg_rewind</application> will overwrite
|
|
the local node's configuration files with the files from the source node,
|
|
so it's advisable to use this option to ensure they are kept.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
|
|
<varlistentry>
|
|
<term><option>--config-archive-dir</option></term>
|
|
<listitem>
|
|
<para>
|
|
Directory to temporarily store configuration files specified with
|
|
<option>--config-files</option>; default: <filename>/tmp</filename>.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
|
|
<varlistentry>
|
|
<term><option>-W/--no-wait</option></term>
|
|
<listitem>
|
|
<para>
|
|
Don't wait for the node to rejoin cluster.
|
|
</para>
|
|
<para>
|
|
If this option is supplied, &repmgr; will restart the node but
|
|
not wait for it to connect to the primary.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
</variablelist>
|
|
</refsect1>
|
|
|
|
<refsect1>
|
|
<title>Configuration file settings</title>
|
|
|
|
<para>
|
|
<itemizedlist spacing="compact" mark="bullet">
|
|
<listitem>
|
|
<simpara>
|
|
<literal>node_rejoin_timeout</literal>:
|
|
the maximum length of time (in seconds) to wait for
|
|
the node to reconnect to the replication cluster (defaults to
|
|
the value set in <literal>standby_reconnect_timeout</literal>,
|
|
60 seconds).
|
|
</simpara>
|
|
<simpara>
|
|
Note that <literal>standby_reconnect_timeout</literal> must be
|
|
set to a value equal to or greater than
|
|
<literal>node_rejoin_timeout</literal>.
|
|
</simpara>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
|
|
</refsect1>
|
|
|
|
<refsect1 id="repmgr-node-rejoin-events">
|
|
<title>Event notifications</title>
|
|
<para>
|
|
A <literal>node_rejoin</literal> <link linkend="event-notifications">event notification</link> will be generated.
|
|
</para>
|
|
</refsect1>
|
|
<refsect1>
|
|
<title>Exit codes</title>
|
|
<para>
|
|
One of the following exit codes will be emitted by <command>repmgr node rejoin</command>:
|
|
</para>
|
|
|
|
<variablelist>
|
|
|
|
<varlistentry>
|
|
<term><option>SUCCESS (0)</option></term>
|
|
<listitem>
|
|
<para>
|
|
The node rejoin succeeded; or if <option>--dry-run</option> was provided,
|
|
no issues were detected which would prevent the node rejoin.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><option>ERR_BAD_CONFIG (1)</option></term>
|
|
<listitem>
|
|
<para>
|
|
A configuration issue was detected which prevented &repmgr; from
|
|
continuing with the node rejoin.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><option>ERR_NO_RESTART (4)</option></term>
|
|
<listitem>
|
|
<para>
|
|
The node could not be restarted.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><option>ERR_REJOIN_FAIL (24)</option></term>
|
|
<listitem>
|
|
<para>
|
|
The node rejoin operation failed.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
</variablelist>
|
|
|
|
</refsect1>
|
|
|
|
<refsect1>
|
|
<title>Notes</title>
|
|
<para>
|
|
Currently <command>repmgr node rejoin</command> can only be used to attach
|
|
a standby to the current primary, not another standby.
|
|
</para>
|
|
<para>
|
|
The node's PostgreSQL instance must have been shut down cleanly. If this was not the
|
|
case, it will need to be started up until it has reached a consistent recovery point,
|
|
then shut down cleanly.
|
|
</para>
|
|
<para>
|
|
In PostgreSQL 13 and later, this will be done automatically
|
|
if the <option>--force-rewind</option> is provided (even if an actual rewind
|
|
is not necessary).
|
|
</para>
|
|
<para>
|
|
With PostgreSQL 12 and earlier, PostgreSQL will need to
|
|
be started and shut down manually; see below for the best way to do this.
|
|
</para>
|
|
<tip>
|
|
<para>
|
|
If <application>PostgreSQL</application> is started in single-user mode and
|
|
input is directed from <filename>/dev/null/</filename>, it will perform recovery
|
|
then immediately quit, and will then be in a state suitable for use by
|
|
<application>pg_rewind</application>.
|
|
<programlisting>
|
|
rm -f /var/lib/pgsql/data/recovery.conf
|
|
postgres --single -D /var/lib/pgsql/data/ < /dev/null</programlisting>
|
|
</para>
|
|
<para>
|
|
Note that <filename>standby.signal</filename> (PostgreSQL 11 and earlier:
|
|
<filename>recovery.conf</filename>) <emphasis>must</emphasis> be removed
|
|
from the data directory for PostgreSQL to be able to start in single
|
|
user mode.
|
|
</para>
|
|
</tip>
|
|
|
|
</refsect1>
|
|
|
|
<refsect1 id="repmgr-node-rejoin-pg-rewind" xreflabel="Using pg_rewind">
|
|
|
|
<title>Using <command>pg_rewind</command></title>
|
|
|
|
<indexterm>
|
|
<primary>pg_rewind</primary>
|
|
<secondary>using with "repmgr node rejoin"</secondary>
|
|
</indexterm>
|
|
|
|
<para>
|
|
<command>repmgr node rejoin</command> can optionally use <command>pg_rewind</command> to re-integrate a
|
|
node which has diverged from the rest of the cluster, typically a failed primary.
|
|
</para>
|
|
<note>
|
|
<para>
|
|
<command>pg_rewind</command> <emphasis>requires</emphasis> that either
|
|
<varname>wal_log_hints</varname> is enabled, or that
|
|
data checksums were enabled when the cluster was initialized. See the
|
|
<ulink url="https://www.postgresql.org/docs/current/app-pgrewind.html"><command>pg_rewind</command> documentation</ulink> for details.
|
|
</para>
|
|
<para>
|
|
Additionally, <varname>full_page_writes</varname> must be enabled; this is the default and
|
|
normally should never be disabled.
|
|
</para>
|
|
</note>
|
|
|
|
<para>
|
|
We strongly recommend familiarizing yourself with <command>pg_rewind</command> before attempting
|
|
to use it with &repmgr;, as while it is an extremely useful tool, it is <emphasis>not</emphasis>
|
|
a "magic bullet" which can resolve all problematic replication situations.
|
|
</para>
|
|
|
|
<para>
|
|
A typical use-case for <command>pg_rewind</command> is when a scenario like the following
|
|
is encountered:
|
|
<programlisting>
|
|
$ repmgr node rejoin -f /etc/repmgr.conf -d 'host=node3 dbname=repmgr user=repmgr' \
|
|
--force-rewind --config-files=postgresql.local.conf,postgresql.conf --verbose --dry-run
|
|
NOTICE: rejoin target is node "node3" (node ID: 3)
|
|
INFO: replication connection to the rejoin target node was successful
|
|
INFO: local and rejoin target system identifiers match
|
|
DETAIL: system identifier is 6652184002263212600
|
|
ERROR: this node cannot attach to rejoin target node 3
|
|
DETAIL: rejoin target server's timeline 2 forked off current database system timeline 1 before current recovery point 0/610D710
|
|
HINT: use --force-rewind to execute pg_rewind</programlisting>
|
|
|
|
Here, <literal>node3</literal> was promoted to a primary while the local node was
|
|
still attached to the previous primary; this can potentially happen during e.g. a
|
|
network split. <command>pg_rewind</command> can re-sync the local node with <literal>node3</literal>,
|
|
removing the need for a full reclone.
|
|
</para>
|
|
|
|
<para>
|
|
To have <command>repmgr node rejoin</command> use <command>pg_rewind</command>,
|
|
pass the command line option <literal>--force-rewind</literal>, which will tell &repmgr;
|
|
to execute <command>pg_rewind</command> to ensure the node can be rejoined successfully.
|
|
</para>
|
|
|
|
<refsect2 id="repmgr-node-rejoin-pg-rewind-config-files" xreflabel="pg_rewind and configuration files">
|
|
|
|
<title><command>pg_rewind</command> and configuration file retention</title>
|
|
|
|
<indexterm>
|
|
<primary>pg_rewind</primary>
|
|
<secondary>configuration file retention</secondary>
|
|
</indexterm>
|
|
|
|
<para>
|
|
Be aware that if <command>pg_rewind</command> is executed and actually performs a
|
|
rewind operation, any configuration files in the PostgreSQL data directory will be
|
|
overwritten with those from the source server.
|
|
</para>
|
|
<para>
|
|
To prevent this happening, provide a comma-separated list of files to retain
|
|
using the <option>--config-file</option> command line option; the specified files
|
|
will be archived in a temporary directory (whose parent directory can be specified with
|
|
<option>--config-archive-dir</option>, default: <filename>/tmp</filename>)
|
|
and restored once the rewind operation is complete.
|
|
</para>
|
|
</refsect2>
|
|
|
|
<refsect2 id="repmgr-node-rejoin-pg-rewind-example" xreflabel="example using repmgr node rejoin and pg_rewind">
|
|
|
|
<title>Example using <command>repmgr node rejoin</command> and <command>pg_rewind</command></title>
|
|
|
|
<indexterm>
|
|
<primary>pg_rewind</primary>
|
|
<secondary>configuration file retention</secondary>
|
|
</indexterm>
|
|
|
|
|
|
<para>
|
|
Example, first using <option>--dry-run</option>, then actually executing the
|
|
<literal>node rejoin command</literal>.
|
|
<programlisting>
|
|
$ repmgr node rejoin -f /etc/repmgr.conf -d 'host=node3 dbname=repmgr user=repmgr' \
|
|
--config-files=postgresql.local.conf,postgresql.conf --verbose --force-rewind --dry-run
|
|
NOTICE: rejoin target is node "node3" (node ID: 3)
|
|
INFO: replication connection to the rejoin target node was successful
|
|
INFO: local and rejoin target system identifiers match
|
|
DETAIL: system identifier is 6652460429293670710
|
|
NOTICE: pg_rewind execution required for this node to attach to rejoin target node 3
|
|
DETAIL: rejoin target server's timeline 2 forked off current database system timeline 1 before current recovery point 0/610D710
|
|
INFO: prerequisites for using pg_rewind are met
|
|
INFO: file "postgresql.local.conf" would be copied to "/tmp/repmgr-config-archive-node2/postgresql.local.conf"
|
|
INFO: file "postgresql.replication-setup.conf" would be copied to "/tmp/repmgr-config-archive-node2/postgresql.replication-setup.conf"
|
|
INFO: pg_rewind would now be executed
|
|
DETAIL: pg_rewind command is:
|
|
pg_rewind -D '/var/lib/postgresql/data' --source-server='host=node3 dbname=repmgr user=repmgr'
|
|
INFO: prerequisites for executing NODE REJOIN are met</programlisting>
|
|
|
|
<note>
|
|
<para>
|
|
If <option>--force-rewind</option> is used with the <option>--dry-run</option> option,
|
|
this checks the prerequisites for using <application>pg_rewind</application>, but is
|
|
not an absolute guarantee that actually executing <application>pg_rewind</application>
|
|
will succeed. See also section <xref linkend="repmgr-node-rejoin-caveats"/> below.
|
|
</para>
|
|
|
|
</note>
|
|
|
|
<programlisting>
|
|
$ repmgr node rejoin -f /etc/repmgr.conf -d 'host=node3 dbname=repmgr user=repmgr' \
|
|
--config-files=postgresql.local.conf,postgresql.conf --verbose --force-rewind
|
|
NOTICE: pg_rewind execution required for this node to attach to rejoin target node 3
|
|
DETAIL: rejoin target server's timeline 2 forked off current database system timeline 1 before current recovery point 0/610D710
|
|
NOTICE: executing pg_rewind
|
|
DETAIL: pg_rewind command is "pg_rewind -D '/var/lib/postgresql/data' --source-server='host=node3 dbname=repmgr user=repmgr'"
|
|
NOTICE: 2 files copied to /var/lib/postgresql/data
|
|
NOTICE: setting node 2's upstream to node 3
|
|
NOTICE: starting server using "pg_ctl -l /var/log/postgres/startup.log -w -D '/var/lib/pgsql/data' start"
|
|
NOTICE: NODE REJOIN successful
|
|
DETAIL: node 2 is now attached to node 3</programlisting>
|
|
</para>
|
|
</refsect2>
|
|
|
|
<refsect2 id="repmgr-node-rejoin-postgresql-94" xreflabel="pg_rewind and PostgreSQL 9.4">
|
|
|
|
<title><command>pg_rewind</command> and PostgreSQL 9.4</title>
|
|
|
|
<indexterm>
|
|
<primary>pg_rewind</primary>
|
|
<secondary>PostgreSQL 9.4</secondary>
|
|
</indexterm>
|
|
|
|
<para>
|
|
<application>pg_rewind</application> is available in PostgreSQL 9.5 and later as part of the core distribution.
|
|
Users of PostgreSQL 9.4 will need to manually install it; the source code is available here:
|
|
<ulink url="https://github.com/vmware/pg_rewind">https://github.com/vmware/pg_rewind</ulink>.
|
|
If the <application>pg_rewind</application>
|
|
binary is not installed in the PostgreSQL <filename>bin</filename> directory, provide
|
|
its full path on the demotion candidate with <option>--force-rewind</option>.
|
|
</para>
|
|
<para>
|
|
Note that building the 9.4 version of <application>pg_rewind</application> requires the PostgreSQL
|
|
source code.
|
|
</para>
|
|
</refsect2>
|
|
</refsect1>
|
|
|
|
<refsect1 id="repmgr-node-rejoin-caveats" xreflabel="Caveats">
|
|
|
|
<title>Caveats when using <command>repmgr node rejoin</command></title>
|
|
|
|
<indexterm>
|
|
<primary>repmgr node rejoin</primary>
|
|
<secondary>caveats</secondary>
|
|
</indexterm>
|
|
|
|
<para>
|
|
<command>repmgr node rejoin</command> attempts to determine whether it will succeed by
|
|
comparing the timelines and relative WAL positions of the local node (rejoin candidate) and primary
|
|
(rejoin target). This is particularly important if planning to use <application>pg_rewind</application>,
|
|
which currently (as of PostgreSQL 12) may appear to succeed (or indicate there is no action
|
|
needed) but potentially allow an impossible action, such as trying to rejoin a standby to a
|
|
primary which is behind the standby. &repmgr; will prevent this situation from occurring.
|
|
</para>
|
|
<para>
|
|
Currently it is <emphasis>not</emphasis> possible to detect a situation where the rejoin target
|
|
is a standby which has been "promoted" by removing <filename>recovery.conf</filename>
|
|
(PostgreSQL 12 and later: <filename>standby.signal</filename>) and restarting it.
|
|
In this case there will be no information about the point the rejoin target diverged
|
|
from the current standby; the rejoin operation will fail and
|
|
the current standby's PostgreSQL log will contain entries with the text
|
|
"<literal>record with incorrect prev-link</literal>".
|
|
</para>
|
|
<para>
|
|
In PostgreSQL 9.5 and earlier, it is <emphasis>not</emphasis> possible to use
|
|
<application>pg_rewind</application> to attach to a target node with a lower
|
|
timeline than the local node.
|
|
</para>
|
|
<para>
|
|
We strongly recommend running <command>repmgr node rejoin</command> with the
|
|
<option>--dry-run</option> option first. Additionally it might be a good idea
|
|
to execute the <application>pg_rewind</application> command displayed by
|
|
&repmgr; with the <application>pg_rewind</application> <option>--dry-run</option>
|
|
option. Note that <application>pg_rewind</application> does not indicate that it
|
|
is running in <option>--dry-run</option> mode.
|
|
</para>
|
|
|
|
<warning>
|
|
<para>
|
|
In all PostgreSQL released before February 2021, <application>pg_rewind</application>
|
|
contains a corner-case bug which affects standbys in a very specific situation.
|
|
</para>
|
|
<para>
|
|
This situation occurs when a standby was shut down <emphasis>before</emphasis> its
|
|
primary node, and an attempt is made to attach this standby to another primary
|
|
in the same cluster (following a "split brain" situation where the standby
|
|
was connected to the wrong primary). In this case, &repmgr; will correctly determine
|
|
that <application>pg_rewind</application> should be executed, however
|
|
<application>pg_rewind</application> incorrectly decides that no action is necessary.
|
|
</para>
|
|
<para>
|
|
In this situation, &repmgr; will report something like:
|
|
<programlisting>
|
|
NOTICE: pg_rewind execution required for this node to attach to rejoin target node 1
|
|
DETAIL: rejoin target server's timeline 3 forked off current database system timeline 2 before current recovery point 0/7019C10</programlisting>
|
|
but when executed, <application>pg_rewind</application> will report:
|
|
<programlisting>
|
|
pg_rewind: servers diverged at WAL location 0/7015540 on timeline 2
|
|
pg_rewind: no rewind required</programlisting>
|
|
and if an attempt is made to attach the standby to the new primary, PostgreSQL logs on the standby
|
|
will contain errors like:
|
|
<programlisting>
|
|
[2020-09-07 15:01:41 UTC] LOG: 00000: replication terminated by primary server
|
|
[2020-09-07 15:01:41 UTC] DETAIL: End of WAL reached on timeline 2 at 0/7015540.
|
|
[2020-09-07 15:01:41 UTC] LOG: 00000: new timeline 3 forked off current database system timeline 2 before current recovery point 0/7019C10</programlisting>
|
|
</para>
|
|
<para>
|
|
Currently it is not possible to resolve this situation using <application>pg_rewind</application>.
|
|
A <ulink url="https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=2b4f3130382fe2f8705863e4d38589d4d69cd695">patch</ulink>
|
|
was submitted and is included in all PostgreSQL versions released in February 2021 or later.
|
|
</para>
|
|
<para>
|
|
As a workaround, start the primary server the standby was previously attached to,
|
|
and ensure the standby can be attached to it. If <application>pg_rewind</application> was actually executed,
|
|
it will have copied in the <filename>.history</filename> file from the target primary server; this must
|
|
be removed. <command>repmgr node rejoin</command> can then be used to attach the standby to the original
|
|
primary. Ensure any changes pending on the primary have propagated to the standby. Then shut down the primary
|
|
server <emphasis>first</emphasis>, before shutting down the standby. It should then be possible to
|
|
use <command>repmgr node rejoin</command> to attach the standby to the new primary.
|
|
</para>
|
|
</warning>
|
|
|
|
</refsect1>
|
|
|
|
<refsect1>
|
|
<title>See also</title>
|
|
<para>
|
|
<xref linkend="repmgr-standby-follow"/>, <xref linkend="repmgr-standby-switchover"/>
|
|
</para>
|
|
</refsect1>
|
|
</refentry>
|