mirror of
https://github.com/EnterpriseDB/repmgr.git
synced 2026-03-23 07:06:30 +00:00
458 lines
19 KiB
XML
458 lines
19 KiB
XML
<refentry id="repmgr-node-rejoin">
|
|
|
|
<indexterm>
|
|
<primary>repmgr node rejoin</primary>
|
|
</indexterm>
|
|
|
|
<refmeta>
|
|
<refentrytitle>repmgr node rejoin</refentrytitle>
|
|
</refmeta>
|
|
|
|
<refnamediv>
|
|
<refname>repmgr node rejoin</refname>
|
|
<refpurpose>rejoin a dormant (stopped) node to the replication cluster</refpurpose>
|
|
</refnamediv>
|
|
|
|
<refsect1>
|
|
<title>Description</title>
|
|
<para>
|
|
Enables a dormant (stopped) node to be rejoined to the replication cluster.
|
|
</para>
|
|
<para>
|
|
This can optionally use <application>pg_rewind</application> to re-integrate
|
|
a node which has diverged from the rest of the cluster, typically a failed primary.
|
|
</para>
|
|
|
|
<tip>
|
|
<para>
|
|
If the node is running and needs to be attached to the current primary, use
|
|
<xref linkend="repmgr-standby-follow"/>.
|
|
</para>
|
|
<para>
|
|
Note <xref linkend="repmgr-standby-follow"/> can only be used for standbys which have not diverged
|
|
from the rest of the cluster.
|
|
</para>
|
|
</tip>
|
|
</refsect1>
|
|
|
|
|
|
<refsect1>
|
|
<title>Usage</title>
|
|
|
|
<para>
|
|
<programlisting>
|
|
repmgr node rejoin -d '$conninfo'</programlisting>
|
|
|
|
where <literal>$conninfo</literal> is the PostgreSQL <literal>conninfo</literal> string of the
|
|
<emphasis>current</emphasis> primary node (or that of any reachable node in the cluster, but
|
|
<emphasis>not</emphasis> the local node). This is so that &repmgr; can fetch up-to-date information
|
|
about the current state of the cluster.
|
|
</para>
|
|
<para>
|
|
<filename>repmgr.conf</filename> for the stopped node *must* be supplied explicitly if not
|
|
otherwise available.
|
|
</para>
|
|
</refsect1>
|
|
|
|
<refsect1>
|
|
|
|
<title>Options</title>
|
|
<variablelist>
|
|
|
|
<varlistentry>
|
|
<term><option>--dry-run</option></term>
|
|
<listitem>
|
|
<para>
|
|
Check prerequisites but don't actually execute the rejoin.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><option>--force-rewind[=/path/to/pg_rewind]</option></term>
|
|
<listitem>
|
|
<para>
|
|
Execute <application>pg_rewind</application>.
|
|
</para>
|
|
<para>
|
|
It is only necessary to provide the <application>pg_rewind</application> path
|
|
if using PostgreSQL 9.4, and <application>pg_rewind</application>
|
|
is not installed in the PostgreSQL <filename>bin</filename> directory.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><option>--config-files</option></term>
|
|
<listitem>
|
|
<para>
|
|
comma-separated list of configuration files to retain after
|
|
executing <application>pg_rewind</application>.
|
|
</para>
|
|
<para>
|
|
Currently <application>pg_rewind</application> will overwrite
|
|
the local node's configuration files with the files from the source node,
|
|
so it's advisable to use this option to ensure they are kept.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
|
|
<varlistentry>
|
|
<term><option>--config-archive-dir</option></term>
|
|
<listitem>
|
|
<para>
|
|
Directory to temporarily store configuration files specified with
|
|
<option>--config-files</option>; default: <filename>/tmp</filename>.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
|
|
<varlistentry>
|
|
<term><option>-W/--no-wait</option></term>
|
|
<listitem>
|
|
<para>
|
|
Don't wait for the node to rejoin cluster.
|
|
</para>
|
|
<para>
|
|
If this option is supplied, &repmgr; will restart the node but
|
|
not wait for it to connect to the primary.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
</variablelist>
|
|
</refsect1>
|
|
|
|
<refsect1>
|
|
<title>Configuration file settings</title>
|
|
|
|
<para>
|
|
<itemizedlist spacing="compact" mark="bullet">
|
|
<listitem>
|
|
<simpara>
|
|
<literal>node_rejoin_timeout</literal>:
|
|
the maximum length of time (in seconds) to wait for
|
|
the node to reconnect to the replication cluster (defaults to
|
|
the value set in <literal>standby_reconnect_timeout</literal>,
|
|
60 seconds).
|
|
</simpara>
|
|
<simpara>
|
|
Note that <literal>standby_reconnect_timeout</literal> must be
|
|
set to a value equal to or greater than
|
|
<literal>node_rejoin_timeout</literal>.
|
|
</simpara>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
|
|
</refsect1>
|
|
|
|
<refsect1 id="repmgr-node-rejoin-events">
|
|
<title>Event notifications</title>
|
|
<para>
|
|
A <literal>node_rejoin</literal> <link linkend="event-notifications">event notification</link> will be generated.
|
|
</para>
|
|
</refsect1>
|
|
<refsect1>
|
|
<title>Exit codes</title>
|
|
<para>
|
|
One of the following exit codes will be emitted by <command>repmgr node rejoin</command>:
|
|
</para>
|
|
|
|
<variablelist>
|
|
|
|
<varlistentry>
|
|
<term><option>SUCCESS (0)</option></term>
|
|
<listitem>
|
|
<para>
|
|
The node rejoin succeeded; or if <option>--dry-run</option> was provided,
|
|
no issues were detected which would prevent the node rejoin.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><option>ERR_BAD_CONFIG (1)</option></term>
|
|
<listitem>
|
|
<para>
|
|
A configuration issue was detected which prevented &repmgr; from
|
|
continuing with the node rejoin.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><option>ERR_NO_RESTART (4)</option></term>
|
|
<listitem>
|
|
<para>
|
|
The node could not be restarted.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><option>ERR_REJOIN_FAIL (24)</option></term>
|
|
<listitem>
|
|
<para>
|
|
The node rejoin operation failed.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
</variablelist>
|
|
|
|
</refsect1>
|
|
|
|
<refsect1>
|
|
<title>Notes</title>
|
|
<para>
|
|
Currently <command>repmgr node rejoin</command> can only be used to attach
|
|
a standby to the current primary, not another standby.
|
|
</para>
|
|
<para>
|
|
The node must have been shut down cleanly; if this was not the case, it will
|
|
need to be manually started (remove any existing <filename>recovery.conf</filename> file first)
|
|
until it has reached a consistent recovery point, then shut down cleanly.
|
|
</para>
|
|
<tip>
|
|
<para>
|
|
If <application>PostgreSQL</application> is started in single-user mode and
|
|
input is directed from <filename>/dev/null/</filename>, it will perform recovery
|
|
then immediately quit, and will then be in a state suitable for use by
|
|
<application>pg_rewind</application>.
|
|
<programlisting>
|
|
rm -f /var/lib/pgsql/data/recovery.conf
|
|
postgres --single -D /var/lib/pgsql/data/ < /dev/null</programlisting>
|
|
</para>
|
|
</tip>
|
|
<para>
|
|
&repmgr; will attempt to verify whether the node can rejoin as-is, or whether
|
|
<command>pg_rewind</command> must be used (see following section).
|
|
</para>
|
|
</refsect1>
|
|
|
|
<refsect1 id="repmgr-node-rejoin-pg-rewind" xreflabel="Using pg_rewind">
|
|
|
|
<title>Using <command>pg_rewind</command></title>
|
|
|
|
<indexterm>
|
|
<primary>pg_rewind</primary>
|
|
<secondary>using with "repmgr node rejoin"</secondary>
|
|
</indexterm>
|
|
|
|
<para>
|
|
<command>repmgr node rejoin</command> can optionally use <command>pg_rewind</command> to re-integrate a
|
|
node which has diverged from the rest of the cluster, typically a failed primary.
|
|
<command>pg_rewind</command> is available in PostgreSQL 9.5 and later as part of the core distribution,
|
|
and can be installed from external sources for PostgreSQL 9.4.
|
|
</para>
|
|
<note>
|
|
<para>
|
|
<command>pg_rewind</command> <emphasis>requires</emphasis> that either
|
|
<varname>wal_log_hints</varname> is enabled, or that
|
|
data checksums were enabled when the cluster was initialized. See the
|
|
<ulink url="https://www.postgresql.org/docs/current/app-pgrewind.html"><command>pg_rewind</command> documentation</ulink> for details.
|
|
</para>
|
|
</note>
|
|
|
|
<para>
|
|
We strongly recommend familiarizing yourself with <command>pg_rewind</command> before attempting
|
|
to use it with &repmgr;, as while it is an extremely useful tool, it is <emphasis>not</emphasis>
|
|
a "magic bullet" which can resolve all problematic replication situations.
|
|
</para>
|
|
|
|
<para>
|
|
A typical use-case for <command>pg_rewind</command> is when a scenario like the following
|
|
is encountered:
|
|
<programlisting>
|
|
$ repmgr node rejoin -f /etc/repmgr.conf -d 'host=node3 dbname=repmgr user=repmgr' \
|
|
--force-rewind --config-files=postgresql.local.conf,postgresql.conf --verbose --dry-run
|
|
INFO: replication connection to the rejoin target node was successful
|
|
INFO: local and rejoin target system identifiers match
|
|
DETAIL: system identifier is 6652184002263212600
|
|
ERROR: this node cannot attach to rejoin target node 3
|
|
DETAIL: rejoin target server's timeline 2 forked off current database system timeline 1 before current recovery point 0/610D710
|
|
HINT: use --force-rewind to execute pg_rewind</programlisting>
|
|
|
|
Here, <literal>node3</literal> was promoted to a primary while the local node was
|
|
still attached to the previous primary; this can potentially happen during e.g. a
|
|
network split. <command>pg_rewind</command> can re-sync the local node with <literal>node3</literal>,
|
|
removing the need for a full reclone.
|
|
</para>
|
|
|
|
<para>
|
|
To have <command>repmgr node rejoin</command> use <command>pg_rewind</command>,
|
|
pass the command line option <literal>--force-rewind</literal>, which will tell &repmgr;
|
|
to execute <command>pg_rewind</command> to ensure the node can be rejoined successfully.
|
|
</para>
|
|
|
|
<refsect2 id="repmgr-node-rejoin-pg-rewind-config-files" xreflabel="pg_rewind and configuration files">
|
|
|
|
<title><command>pg_rewind</command> and configuration file retention</title>
|
|
|
|
<indexterm>
|
|
<primary>pg_rewind</primary>
|
|
<secondary>configuration file retention</secondary>
|
|
</indexterm>
|
|
|
|
<para>
|
|
Be aware that if <command>pg_rewind</command> is executed and actually performs a
|
|
rewind operation, any configuration files in the PostgreSQL data directory will be
|
|
overwritten with those from the source server.
|
|
</para>
|
|
<para>
|
|
To prevent this happening, provide a comma-separated list of files to retain
|
|
using the <option>--config-file</option> command line option; the specified files
|
|
will be archived in a temporary directory (whose parent directory can be specified with
|
|
<option>--config-archive-dir</option>, default: <filename>/tmp</filename>)
|
|
and restored once the rewind operation is complete.
|
|
</para>
|
|
</refsect2>
|
|
|
|
<refsect2 id="repmgr-node-rejoin-pg-rewind-example" xreflabel="example using repmgr node rejoin and pg_rewind">
|
|
|
|
<title>Example using <command>repmgr node rejoin</command> and <command>pg_rewind</command></title>
|
|
|
|
<indexterm>
|
|
<primary>pg_rewind</primary>
|
|
<secondary>configuration file retention</secondary>
|
|
</indexterm>
|
|
|
|
|
|
<para>
|
|
Example, first using <option>--dry-run</option>, then actually executing the
|
|
<literal>node rejoin command</literal>.
|
|
<programlisting>
|
|
$ repmgr node rejoin -f /etc/repmgr.conf -d 'host=node3 dbname=repmgr user=repmgr' \
|
|
--config-files=postgresql.local.conf,postgresql.conf --verbose --force-rewind --dry-run
|
|
INFO: replication connection to the rejoin target node was successful
|
|
INFO: local and rejoin target system identifiers match
|
|
DETAIL: system identifier is 6652460429293670710
|
|
NOTICE: pg_rewind execution required for this node to attach to rejoin target node 3
|
|
DETAIL: rejoin target server's timeline 2 forked off current database system timeline 1 before current recovery point 0/610D710
|
|
INFO: prerequisites for using pg_rewind are met
|
|
INFO: file "postgresql.local.conf" would be copied to "/tmp/repmgr-config-archive-node2/postgresql.local.conf"
|
|
INFO: file "postgresql.replication-setup.conf" would be copied to "/tmp/repmgr-config-archive-node2/postgresql.replication-setup.conf"
|
|
INFO: pg_rewind would now be executed
|
|
DETAIL: pg_rewind command is:
|
|
pg_rewind -D '/var/lib/postgresql/data' --source-server='host=node3 dbname=repmgr user=repmgr'
|
|
INFO: prerequisites for executing NODE REJOIN are met</programlisting>
|
|
|
|
<note>
|
|
<para>
|
|
If <option>--force-rewind</option> is used with the <option>--dry-run</option> option,
|
|
this checks the prerequisites for using <application>pg_rewind</application>, but is
|
|
not an absolute guarantee that actually executing <application>pg_rewind</application>
|
|
will succeed. See also section <xref linkend="repmgr-node-rejoin-caveats"/> below.
|
|
</para>
|
|
|
|
</note>
|
|
|
|
<programlisting>
|
|
$ repmgr node rejoin -f /etc/repmgr.conf -d 'host=node3 dbname=repmgr user=repmgr' \
|
|
--config-files=postgresql.local.conf,postgresql.conf --verbose --force-rewind
|
|
NOTICE: pg_rewind execution required for this node to attach to rejoin target node 3
|
|
DETAIL: rejoin target server's timeline 2 forked off current database system timeline 1 before current recovery point 0/610D710
|
|
NOTICE: executing pg_rewind
|
|
DETAIL: pg_rewind command is "pg_rewind -D '/var/lib/postgresql/data' --source-server='host=node3 dbname=repmgr user=repmgr'"
|
|
NOTICE: 2 files copied to /var/lib/postgresql/data
|
|
NOTICE: setting node 2's upstream to node 3
|
|
NOTICE: starting server using "pg_ctl -l /var/log/postgres/startup.log -w -D '/var/lib/pgsql/data' start"
|
|
NOTICE: NODE REJOIN successful
|
|
DETAIL: node 2 is now attached to node 3</programlisting>
|
|
</para>
|
|
</refsect2>
|
|
</refsect1>
|
|
|
|
<refsect1 id="repmgr-node-rejoin-caveats" xreflabel="Caveats">
|
|
|
|
<title>Caveats when using <command>repmgr node rejoin</command></title>
|
|
|
|
<indexterm>
|
|
<primary>repmgr node rejoin</primary>
|
|
<secondary>caveats</secondary>
|
|
</indexterm>
|
|
|
|
<para>
|
|
<command>repmgr node rejoin</command> attempts to determine whether it will succeed by
|
|
comparing the timelines and relative WAL positions of the local node (rejoin candidate) and primary
|
|
(rejoin target). This is particularly important if planning to use <application>pg_rewind</application>,
|
|
which currently (as of PostgreSQL 12) may appear to succeed (or indicate there is no action
|
|
needed) but potentially allow an impossible action, such as trying to rejoin a standby to a
|
|
primary which is behind the standby. &repmgr; will prevent this situation from occurring.
|
|
</para>
|
|
<para>
|
|
Currently it is <emphasis>not</emphasis> possible to detect a situation where the rejoin target
|
|
is a standby which has been "promoted" by removing <filename>recovery.conf</filename>
|
|
(PostgreSQL 12 and later: <filename>standby.signal</filename>) and restarting it.
|
|
In this case there will be no information about the point the rejoin target diverged
|
|
from the current standby; the rejoin operation will fail and
|
|
the current standby's PostgreSQL log will contain entries with the text
|
|
"<literal>record with incorrect prev-link</literal>".
|
|
</para>
|
|
<para>
|
|
We strongly recommend running <command>repmgr node rejoin</command> with the
|
|
<option>--dry-run</option> option first. Additionally it might be a good idea
|
|
to execute the <application>pg_rewind</application> command displayed by
|
|
&repmgr; with the <application>pg_rewind</application> <option>--dry-run</option>
|
|
option. Note that <application>pg_rewind</application> does not indicate that it
|
|
is running in <option>--dry-run</option> mode.
|
|
</para>
|
|
|
|
<warning>
|
|
<para>
|
|
In all current PostgreSQL versions (as of September 2020), <application>pg_rewind</application>
|
|
contains a corner-case bug which affects standbys in a very specific situation.
|
|
</para>
|
|
<para>
|
|
This situation occurs when a standby was shut down <emphasis>before</emphasis> its
|
|
primary node, and an attempt is made to attach this standby to another primary
|
|
in the same cluster (following a "split brain" situation where the standby
|
|
was connected to the wrong primary). In this case, &repmgr; will correctly determine
|
|
that <application>pg_rewind</application> should be executed, however
|
|
<application>pg_rewind</application> incorrectly decides that no action is necessary.
|
|
</para>
|
|
<para>
|
|
In this situation, &repmgr; will report something like:
|
|
<programlisting>
|
|
NOTICE: pg_rewind execution required for this node to attach to rejoin target node 1
|
|
DETAIL: rejoin target server's timeline 3 forked off current database system timeline 2 before current recovery point 0/7019C10</programlisting>
|
|
but when executed, <application>pg_rewind</application> will report:
|
|
<programlisting>
|
|
pg_rewind: servers diverged at WAL location 0/7015540 on timeline 2
|
|
pg_rewind: no rewind required</programlisting>
|
|
and if an attempt is made to attach the standby to the new primary, PostgreSQL logs on the standby
|
|
will contain errors like:
|
|
<programlisting>
|
|
[2020-09-07 15:01:41 UTC] LOG: 00000: replication terminated by primary server
|
|
[2020-09-07 15:01:41 UTC] DETAIL: End of WAL reached on timeline 2 at 0/7015540.
|
|
[2020-09-07 15:01:41 UTC] LOG: 00000: new timeline 3 forked off current database system timeline 2 before current recovery point 0/7019C10</programlisting>
|
|
</para>
|
|
<para>
|
|
Currently it is not possible to resolve this situation using <application>pg_rewind</application>.
|
|
A <ulink url="https://www.postgresql.org/message-id/flat/CABvVfJU-LDWvoz4-Yow3Ay5LZYTuPD7eSjjE4kGyNZpXC6FrVQ@mail.gmail.com">patch</ulink>
|
|
has been submitted and will hopefully be included in a forthcoming PostgreSQL minor release.
|
|
</para>
|
|
<para>
|
|
As a workaround, start the primary server the standby was previously attached to,
|
|
and ensure the standby can be attached to it. If <application>pg_rewind</application> was actually executed,
|
|
it will have copied in the <filename>.history</filename> file from the target primary server; this must
|
|
be removed. <command>repmgr node rejoin</command> can then be used to attach the standby to the original
|
|
primary. Ensure any changes pending on the primary have propogated to the standby. Then shut down the primary
|
|
server <emphasis>first</emphasis>, before shutting down the standby. It should then be possible to
|
|
use <command>repmgr node rejoin</command> to attach the standby to the new primary.
|
|
</para>
|
|
</warning>
|
|
|
|
</refsect1>
|
|
|
|
<refsect1>
|
|
<title>See also</title>
|
|
<para>
|
|
<xref linkend="repmgr-standby-follow"/>
|
|
</para>
|
|
</refsect1>
|
|
</refentry>
|