Make the code match the documentation.
As pointed out by GitHub user phyber (#142).
Also various other minor improvements to error reporting during
config file parsing.
The difference between this and establish_db_connection() is that
it outputs any connection failure as a [NOTICE] rather than an
[ERROR]; it's intended for use in e.g. polling a server to wait
for it to come up/go down, while preventing [ERROR] log lines
which may cause confusion.
There's no compelling reason to require "archive_mode" to be enabled
for streaming replication. It is of course a good idea to archive WAL
using e.g. barman ( http://www.pgbarman.org/ ) as part of a comprehensive
backup strategy, but repmgr and streaming replication work fine without
it.
Per GitHub #141.
Also revise the configuration check for "archive_command" to be
triggered only when "archive_mode" is not "off", as from PostgreSQL
9.5 onwards "archive_mode" can also be "on" or "always".
Updated makefile for deb creation
There's still a bug in the Version field of the Control file (it shouldn't have a 'v' in front of the version).
Will fix that immediately after.
A fix for this was introduced with commit ee9270fe8d
and removed in 4f1c67a1bf.
Refactor the original fix to simply omit attempting to write an invalid entry
to the monitoring table.
Calculate the width of the "Name" and "Upstream" columns dynamically.
Based on pull request #135 by sengaya, edited and modified by myself
to include a psql-like separator line.
removing it.
Basically, on startup the standby will start receiving again from the
begining of the WAL and so received will be lower then applied.
A proper code is needed to make sure the standby is still following the
correct master (as per node information)
used in the cluster.
Main issue was that if the local repmgrd was not able to connect locally,
it would set the local node as failed (active = false). This is fine, because
we actually don't know if the node is active (actually, it's not active ATM)
so it's best to keep it out of the cluster.
The problem is that if the postgres service comes back up, and is able to
recover by it self, then we should ack that fact and set it as active.
There was another issue related with repmgrd being terminated if the postgres
service was downs. This is not the correct thing to do: we should keep
trying to connect to the local standby.
used in the cluster.
Main issue was that if the local repmgrd was not able to connect locally,
it would set the local node as failed (active = false). This is fine, because
we actually don't know if the node is active (actually, it's not active ATM)
so it's best to keep it out of the cluster.
The problem is that if the postgres service comes back up, and is able to
recover by it self, then we should ack that fact and set it as active.
There was another issue related with repmgrd being terminated if the postgres
service was downs. This is not the correct thing to do: we should keep
trying to connect to the local standby.
Perform a switchover by:
- stopping current primary node
- promoting this standby node to primary
- forcing previous primary node to follow this node
Caveats:
- repmgrd must not be running, otherwise it may
attempt a failover
(TODO: find some way of notifying repmgrd of planned
activity like this)
- currently only set up for two-node operation; any other
standbys will probably become downstream cascaded standbys
of the old primary once it's restarted
- as we're executing repmgr remotely (on the old primary),
we'll need the location of its configuration file; this
can be provided explicitly with -C/--remote-config-file,
otherwise repmgr will look in default locations on the
remote server
- this does not yet support "rewinding" stopped nodes
which will be unable to catch up with the primary
TODO:
- update help, docs
- make connection test timeouts/intervals configurable
Anyone needing them, particularly in a replication context, should
know what they're doing anyway.
See also: http://www.postgresql.org/docs/current/interactive/sql-createindex.html#AEN74175
"Also, changes to hash indexes are not replicated over streaming or file-based
replication after the initial base backup, so they give wrong answers to
queries that subsequently use them. For these reasons, hash index use is presently
discouraged."
Also refactor configuration file handling while we're at it.
Previously a configuration file would be ignored if it couldn't
be opened, however that is now treated as an error.
When parsing command line arguments in check_parameters_for_action(),
create warnings for paramters supplied but not required (e.g. -D/--data-dir
for MASTER REGISTER), rather than fail with error(s), as the
presence of the parameters won't cause any problems.
Errors will still be raised for required-but-missing parameters, of course.
* repmgr: add explicit --log-level flag, repurpose --verbose flag to
show extra detailed/repetitive output only (see item below too)
-> e0cbdd5b31
* debug output: show some repetitive output only if --verbose flag set to prevent
excessive log growth
-> 8ab1901a93
This should make it more likely that the actual primary is first
in the retrieved list, reducing the number of connections to
other nodes in the cluster which need to be made.
repmgr and particularly repmgrd currently produce substantial
amounts of log output. Much of this is only useful when troubleshooting
or debugging.
Previously the -v/--verbose option just forced the log level to
INFO. With repmgrd this is pretty pointless - just set the log
level in the configuration file. With repmgr the configuration
file can be overriden by the new -L/--log-level option.
-v/--verbose now provides an additional, chattier/pedantic level
of logging ("Opening *this* logfile", "Executing *this* query",
"running in *this* loop") which is helpful for understanding
repmgr/repmgrd's behaviour, particularly for troubleshooting.
What additional verbose logging is generated will of course a
also depends on the log level set, so e.g. someone trying to
work out which configuration file is actually being opened
can use '--log-level=INFO --verbose' without being bothered
by an avalanche of extra verbose debugging output.
-t/--terse option will silence certain non-essential output, at
the moment any HINTs.
Note that -v/--verbose and -t/--terse are not mutually exclusive
(suggestions for better names welcome).
When cloning a server without this option, and pg_start_backup() takes time
to complete, repmgr appears to hang and give no indication of what may
or may not be happening. The hint provides an explanation for any
delay and possible action which could be taken to mitigate it.
There are a few places where additional hints are written as log
output, usually LOG_NOTICE. Create an explicit function to provide
hints in a standardized manner; by storing the log level of the
previous logger call, we can ensure the hint is only displayed when
the log message itself would be.
Part of an ongoing effort to better control repmgr's logging output.
Related to Github #127.
- use the previously introduced repmgr_atoi() function to parse
integers better
- collate all detected errors and output as a list, rather than
failing on the first error.
Also change "master" to "primary" in the comments for consistency
with main PostgreSQL terminology. We'll need to add aliases
for the configuration parameters at some point...
This was erroneously rendered as --dest-dir. This bugfeature seems
to have been around for a long time; however as the only way anyone
could know of the existence of --dest-dir is by reading the source
code, we can safely remove it.
Pointed out by Github user Jehan-Guillaume (ioguix) de Rorthais.
"Primary" is the term preferred in the PostgreSQL documentation, so
we should at least support it.
Practically this means it's possible to write "rempgr primary register"
in place of "repmgr master register".
The next feature-release should replace "master" with "primary" in
the documentation and log messages.
Per gripe in Github #112.
9.5 introduces the tablespace_map file, which is created on the upstream
node while a backup is running. We need to overwrite this with the
provided values.
Note that we only write explicitly-provided values to the tablespace_map
file, however the existing symlinks for non-specified tablespaces
will have been copied anyway.
Fixes Github #119.
Registering a master creates the schema, but it may be desirable
to forcibly reregister a master without deleting the schema, so
uncouple the dependency.
Also ensure schema creation is atomic by wrapping it in a transaction.
Per GitHub issue #49.
In some cases it is desirable to remove repmgr's handling of a standby, perhaps
because that standby is to be removed altogether. With no UNREGISTER command,
one had to manually delete the corresponding row from repl_nodes.
repmgrd correctly updates ID of the upstream node after automatic
failover, but repmgr was not doing that for manual failvers.
This moves the existing function to dbutils and modifies it so that
it does not rely on global variables with configuration (available
just in repmgrd).
This should fix issue #67 (hopefully, haven't done much testing).
If no configuration file provided, also check default Postgres
sysconfig dir.
It would also be useful to check the configuration directory
provided by the RPM/DEB packages, not sure if that's programmatically
feasible.
Currently repmgr/repmgrd will only accept these as valid when
provided as the first command line option, however it's possible
a user will want to get the output of those options by adding
them to the end of a previously inputted command.
Note that after the first of these options is encountered, the
program will terminate and not process any other options. This
is consistent with psql's behaviour
Per GitHub issue #107 from Sébastien Gross.
If freopen() fails, stderr is diverted to an undisclosed location
and it's not clear what is going on.
Also add an explicit notice announcing our intention to divert
logging output to a file.
Per #105.
Note that it might make sense to disable logfile output when
running the repmgr command line client as normally you'd expect
immediate feedback.
repmgr does not require explicit provision of the target data
directory when cloning a standby (it defaults to the same directory
as on the master). However this is a required option for pg_basebackup
which was only being provided if repmgr's -D/--data-dir option was
set, so ensure we always provide whatever repmgr is using.
Per report from Martín.
The main change is that now check_connection requires a conninfo
parameter, and the connection object has type (PGconn **) so it can be
replaced by check_connection if needed.
The bug was caused by the fact that the first failure resulted in
*conn == NULL, so that subsequent checks of the upstream connection
were failing irrespectively of the actual state of the upstream node.
Now, when *conn == NULL, check_connection will use conninfo to
establish a new connection and place it into *conn. We introduce a new
INTERNAL_ERROR code for the case when they are both NULL.
In passing, we also reworded a confusing error message, distinguishing
a timeout from the actual elapsed time.
Ignore blank lines which consist of whitespace. Per issue #71 in
GitHub.
This fix also improves comment handling and will treat lines with
whitespace before the '#' character as whitespace.
When using the default pg_basebackup method to clone a standby
together with the `--ignore-external-config-files` option, there's
no need to test for a working SSH connection - which may not
be desirable in many use-cases anyway.
Per issue #64 in GitHub.
In some circumstances (primarily when executing `repmgr standby
clone`) the `repmgr.conf` file is not mandated. However this means
the repmgr schema is not known, and any attempt to create an
event record will result in a log warning, which may cause
confusion as to the success of the operation.
It might be better to mandate providing `repmgr.conf` in all
circumstances.
Per report in https://github.com/2ndQuadrant/repmgr/issues/53 .
Provide the master connection if available, and if not enable
create_event_record() to skip trying to write to the database,
but execute the notification program if defined.
- rename --min-recovery-apply-delay to --recovery-min-apply-delay
- ensure server version is 9.4 or later before writing
recovery_min_apply_delay to recovery.conf.
This fixes changes introduced in 653e11c2a7
(the parameter was subsequently renamed).
Also reallocate the '-r' parameter to --rsync-only, which is probably
more useful.
Previous check for the master host was ineffective. We'd be better off explicitly
requiring at least hostname, database and usernames for the master rather
than relying on whatever defaults were in place when STANDBY CLONE is
run, especially as dbname and username are used in recovery.conf.
Event records are advisory and non-critical; there may be paths
where we're not able to connect to a valid master and hence unable
to write a record. (If an 'event_notification_command' is defined,
this should serve as a backup notification of the event).
Command to be executed each time an event is logged.
Following formatting sequences will be interpolated:
%e - event type
%d - description
%s - success (1 or 0)
%t - timestamp
Avoid repeating "it needs", and explain that the superuser password it
needs is for the witness database (and not a sudo password, or a
superuser on the production database).
With 'witness server' in most places.
When defining the nodes, I went with 'server used as a witness', since
'**witness** is the hostname ... of the witness server' sounded like a
tautology. The longer phrasing isn't really any different, but it
sounds better to me ;).
And some other minor tweaks to this paragraph. I split the
witness-repmgrd bit off into its own sentence, since that's pretty
independent of the voting business.
No need to prefix each line with the program name; this was pretty
inconsistent anyway. The only place where log output needs to identify
the outputting program is when syslog is being used, which is done
anyway.
Daemonizing changes the current working directory to '/',
which breaks configuration file parsing if the file is in
the previous working directory and provided without an
explicit path.
Also it makes general sense to parse the configuration file
before daemonizing.
Daemonizing changes the current working directory to '/',
which breaks configuration file parsing if the file is in
the previous working directory and provided without an
explicit path.
Also it makes general sense to parse the configuration file
before daemonizing.
On 9.4 we have logical decoding, which introduced a new wal_level called
logical. This level includes all the previous ones, so you can run a
hot_standby if wal_level = logical, because the relevant information for
hot_standby will be there, plus other information needed for logical
decoding.
We fix this be adding a second check when wal_level is not hot_standby.
This makes keeping track of events such as failovers
much easier. Note that this is for convenience and is
not a foolproof auditing log.
Sample output:
repmgr_db=# SELECT * from repmgr_test.repl_events ;
node_id | event | successful | event_timestamp | details
---------+--------------------------+------------+-------------------------------+----------------------------------------------------------
1 | master_register | t | 2015-03-06 14:14:08.196636+09 |
2 | standby_clone | t | 2015-03-06 14:14:17.660768+09 | Backup method: pg_basebackup; --force: N
2 | standby_register | t | 2015-03-06 14:14:18.762222+09 |
4 | witness_create | t | 2015-03-06 14:14:22.072815+09 |
3 | standby_clone | t | 2015-03-06 14:14:23.524673+09 | Backup method: pg_basebackup; --force: N
3 | standby_register | t | 2015-03-06 14:14:24.620161+09 |
2 | repmgrd_start | t | 2015-03-06 14:14:29.639096+09 |
3 | repmgrd_start | t | 2015-03-06 14:14:29.641489+09 |
4 | repmgrd_start | t | 2015-03-06 14:14:29.648002+09 |
2 | standby_promote | t | 2015-03-06 14:15:01.956737+09 | Node 2 was successfully be promoted to master
2 | repmgrd_failover_promote | t | 2015-03-06 14:15:01.964771+09 | Node 2 promoted to master; old master 1 marked as failed
3 | repmgrd_failover_follow | t | 2015-03-06 14:15:07.228493+09 | Node 3 now following new upstream node 2
(12 rows)
Per e.g. this complaint:
https://groups.google.com/d/msg/repmgr/a-SMIQFGDBs/rgRH3p4ZPgYJ
Before:
$ repmgr -f /etc/postgres/repmgr.conf cluster status
repmgr: Replicator manager
Try "repmgr --help" for more information.
after:
$ repmgr -f /etc/postgres/repmgr.conf cluster status
repmgr: Replication manager
[ERROR] Unknown server command 'status'
Try "repmgr --help" for more information.
This involves mainly abstracting the functions which copy
and create records from repmgr.c to dbutils.c, as they need
to be shared between repmgr and repmgrd.
Per issue noted here:
https://groups.google.com/forum/#!topic/repmgr/v5nu1Xwf6X0
This makes it practical to take a back with rsync where tablespaces
require remapping. It also makes tablespace remapping possible for
9.3, where pg_basebackup does not support this option.
Sometimes it's desirable to re-sync a "stale" data directory
on a standby, rather than start from scratch with pg_basebackup().
This re-adds the rsync code from the 2.x series, with some
modifications.
TODO: tablespace support.
If the user don't put that option in rsync_options using of "--force"
could be unsafe.
While the probability of failures because of this are low they aren't
zero.
If the user don't put that option in rsync_options using of "--force"
could be unsafe.
While the probability of failures because of this are low they aren't
zero.
"pg_basebackup_options"
Enable custom options to be passed to pg_basebackup
(e.g. --max-rate, --checkpoint, --xlogdir)
"tablespace_mapping"
Analogue to pg_basebackup's (9.4 and later) -T/--tablespace-mapping
option.
Tablespace mapping could also be passed via "pg_basebackup_options",
however by providing a separate parameter it makes the configuration
file easier to read and allows us to verify the specified tablespaces
exist (pg_basebackup won't do this, which can lead to undesired
behaviour, i.e. attempting to create the tablespace in the original
path).
Previously, the pg_bindir parameter was mandatory and could only be
provided in the repmgr.conf file, which was leading to the slightly
bizarre situation that e.g. for "clone standby", repmgr was complaining
that it didn't want the configuration file when it actually did.
pg_bindir is now optional - if not provided, it will use the default
path. It can be provided in the repmgr.conf file, or as a command
line parameter; the latter overrides the former.
Previous behaviour was somewhat counterintuitive, with an error
message being logged if no configuration file provided or found,
even though this is not actually an error.
Configuration files now handled like this:
- if a configuration file is explicitly provided (-f), error out
if not found.
- if no configuration file explicitly provided, attempt to open
default configuration file; if this does not exist, log
notice and continue with default values.
Also, for 9.4 and later add a hint about replication slot usage
if 'use_replication_slots' not set.
If the node is a cascaded standby and the primary fails, `primary_conn`
will not be updated automatically; when writing monitoring info,
ensure we connect to the current primary.
Attempt to attach to the next available upstream node, otherwise
quit monitoring. We'll need to add further options for failover
scenarios, including attempting to attach to another node,
shutting down the server completely etc.
To handle cascaded replication we're going to have to keep track
of each node's upstream node. Also enumerate the node type
("primary", "standby" or "witness") and mark if active.
Consolidate the configuration checks carried out when cloning
a standby, and enable them to be run separately to check the
required configuration without running STANDBY CLONE multiple times.
Merge basically identical checks in multiple places into a single
function. Only doing this in repmgr, as repmgrd only performs
the version check in one place and we'd need to somehow pass progname
to the function as well, which isn't worth the effort.
Use the reported `server_version_num` integer for version number
detection and comparison. This makes it easier to set an arbitrary
minimum supported version (rather than "9.0 or later") as well
as future-proofing for 10.x and later.
Previously repmgr passed the -W flag to initdb, which forced
manual input of a password; this option removes the -W flag
to make repetitive testing easier.
Previously the witness database creation code was hard-coding the
username 'postgres' when accessing the previously initialised database.
However initdb was not passed any explicit username, meaning the
default database superuser name was the same as the user running
repmgr.
With this patch, a superuser user name (default: postgres) will
be passed to initdb.
Per report by eggyknap [1]
[1] https://github.com/2ndQuadrant/repmgr/issues/38
we have to assume it's the DEFAULT_MASTER_PORT.
This was not done, so we added a check to see if it has a value that is
usable, else we use DEFAULT_MASTER_PORT.
Using '--exclude=dirname/*' to explicitly specify directories whose contents
should not be copied. This will result in empty directories being created
on the destination if they exist on the source, but that's not a problem as
they are needed anyway.
Previously the generated rsync command contained '--exclude=pg_log*', which
will break replication on 9.5 as the wildcard expansion prevents the
'pg_logical' directory from being copied.
While reading node entries from master use a separate PGresult when inserting into witness.
Witness monitoring supplies a null value for 'last_apply_time'.
repmgr repl_status had the column time_lag which was documented to be
the time a standby is behind master. In fact it only works like this
when viewed on the standby and not on the master: there it only was the
time of the last status update. We dropped that column and replaced it
by a new column „communication_time_lag“ which is the content of the
repl_status column on the master. On the standby we contain the time of
the last update in shared mem though refer always to the correct time
nonetheless where repl_status is queried. We also added a new column,
„replication_time_lag“, which refers to the apply delay.
wait_connection_availability() did take at least 2 seconds per call in
the old incarnation. Now we may finish a call without any sleep at all
when the result is already ready at the time called
We had the problem that the log file appeared empty for a long time due
to file buffers. Thus we call fflush() after every log message so the
log file gets written out to disk quickly
Refactoring part: we now use a function to generate the PID
file. Sophistication: we now check if the PID contained in the file is a
valid PID. We ignore the file if it doesn't.
after the setsid() we are the process leader. And as a process leader we
are able to open a new terminal, even if we currently don't own one. So
we do another fork and do not call setsid() and not become a process
leader to avoid that.
when repmgr_funcs is not pre-loaded `repmgr_update_standby_location()`
will return false and `repmgr_get_last_standby_location()` will return
an empty string. Thus we may end in an endless loop. To avoid that we fail.
When not flushing and fsync()ing it the output may be garbled due to
concurrent writes to the file (system() spawns a child process with
stdin/stdout/stderr inherited from it's parent)
sleep_monitor replaces the old SLEEP_MONITOR define and makes it
configurable; this is the interval in which we monitor
sleep_delay replaces the old sleep(300) when waiting for the master to
recover.
This option forks the process and generates a new session. This
effectively detaches it from the shell. Don't forget to redirect stderr
or use syslog for logging!
We have to remove our own node before we check for a new master if force
is set; else master register would fail on the second time since there
already is a master (ourselves), even if we specify -F
This could leave the database in a locked state (pg_start_backup()).
And since all calls to create_pgdir() handle the return value correctly
we simply replace the exit() by a return false
When deciding which node will be the new master, we should get the
nodes in a well defined order otherwise two standbys could process
nodes with the same priority in different order and end up with
a two master situation.
Because PQcancel() establish a new synchronous connection to the
database, if it fails it means something wrong has happenned with
master. So instead of just ignore the failure, CancelQuery() now
reports a failure condition so we can detect master's death in
that situation.
This is very important specially when only postmaster crashes but
other children/backend connections are still there. Because the
children connection won't fail and CancelQuery() failure is our
only indication of something wrong happenning.
Currently we just ignore the PQcancel() failure which leads us to
a situation in which we just loop forever
trying to cancel the async query.
Reported by: Martin Euser <martin.euser@nl.abnamro.com>
Problem analyzed and bug spotted by: Andres Freund <andres@2ndquadrant.com>
Patch by: Jaime Casanova <jaime@2ndquadrant.com>
voting procedure.
This is because of a race condition inside CheckPrimaryConnection().
This has independently reported by Alex Railean and Dumitru, and Frank Jördens.
Analyzed and fixed by Cédric Villemain.
The fix have been verified to work by Frank
xlog records from master, so it standby should use pg_last_xlog_receive_location()
to report their positions. This solves a possible situation in which
a standby that is considered as new master when promoted is no longer
the best option.
This is important for autofailover to do the right thing when
standbys detected master death at different times.
While this is a new option, seems important for the autofailover
to work properly so i will consider the lack of it a bug and
will backpatch to 2.0 where autofailover was introduced.
For gripe from Alex Railean, about a standby not finding the new
master because the new master hasn't finish promoting.
two nodes with different prorities are equally good to be promoted
the second one (with a lower priority, considering them
in descending order) will win.
Per report from Brailean Dumitru
get a warning for an uninitialized variable.
Also, define InvalidXLogRecPtr. We don't really need it but using
it make the initialization future proof (considering that in 9.3
XLogRecPtr will change its structure).
We need to add an #include and make it use a different path for the
"true" binary.
Maybe we need to make this changes for all BSD systems but having no
evidence of that i prefer to make this only for systems with __FreeBSD__
**cluster* is the name of the current replication.
**node* is the number of the current node (1, 2 or 3 in the current example).
**node_name* is an identifier for every node.
**conninfo* is used to connect to the local PostgreSQL server (where the configuration file is) from any node. In the witness server configuration it is needed to add a 'port=5499' to the conninfo.
**master_response_timeout* is the maximum amount of time we are going to wait before deciding the master has died and start failover procedure.
**reconnect_attempts* is the number of times we will try to reconnect to master after a failure has been detected and before start failover procedure.
**reconnect_interval* is the amount of time between retries to reconnect to master after a failure has been detected and before start failover procedure.
**failover* configure behavior : *manual* or *automatic*.
**promote_command* the command executed to do the failover (including the PostgreSQL failover itself). The command must return 0 on success.
**follow_command* the command executed to address the current standby to another Master. The command must return 0 on success.
It needs information to connect to the master to copy the configuration of the cluster, also it needs to know where it should initialize it's own $PGDATA.
As part of the procees it also ask for the superuser password so it can connect when needed.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
