Perform a switchover by:
- stopping current primary node
- promoting this standby node to primary
- forcing previous primary node to follow this node
Caveats:
- repmgrd must not be running, otherwise it may
attempt a failover
(TODO: find some way of notifying repmgrd of planned
activity like this)
- currently only set up for two-node operation; any other
standbys will probably become downstream cascaded standbys
of the old primary once it's restarted
- as we're executing repmgr remotely (on the old primary),
we'll need the location of its configuration file; this
can be provided explicitly with -C/--remote-config-file,
otherwise repmgr will look in default locations on the
remote server
- this does not yet support "rewinding" stopped nodes
which will be unable to catch up with the primary
TODO:
- update help, docs
- make connection test timeouts/intervals configurable
This should make it more likely that the actual primary is first
in the retrieved list, reducing the number of connections to
other nodes in the cluster which need to be made.
Registering a master creates the schema, but it may be desirable
to forcibly reregister a master without deleting the schema, so
uncouple the dependency.
Also ensure schema creation is atomic by wrapping it in a transaction.
Per GitHub issue #49.
repmgrd correctly updates ID of the upstream node after automatic
failover, but repmgr was not doing that for manual failvers.
This moves the existing function to dbutils and modifies it so that
it does not rely on global variables with configuration (available
just in repmgrd).
This should fix issue #67 (hopefully, haven't done much testing).
In some circumstances (primarily when executing `repmgr standby
clone`) the `repmgr.conf` file is not mandated. However this means
the repmgr schema is not known, and any attempt to create an
event record will result in a log warning, which may cause
confusion as to the success of the operation.
It might be better to mandate providing `repmgr.conf` in all
circumstances.
Per report in https://github.com/2ndQuadrant/repmgr/issues/53 .
Provide the master connection if available, and if not enable
create_event_record() to skip trying to write to the database,
but execute the notification program if defined.
Command to be executed each time an event is logged.
Following formatting sequences will be interpolated:
%e - event type
%d - description
%s - success (1 or 0)
%t - timestamp
No need to prefix each line with the program name; this was pretty
inconsistent anyway. The only place where log output needs to identify
the outputting program is when syslog is being used, which is done
anyway.
This makes keeping track of events such as failovers
much easier. Note that this is for convenience and is
not a foolproof auditing log.
Sample output:
repmgr_db=# SELECT * from repmgr_test.repl_events ;
node_id | event | successful | event_timestamp | details
---------+--------------------------+------------+-------------------------------+----------------------------------------------------------
1 | master_register | t | 2015-03-06 14:14:08.196636+09 |
2 | standby_clone | t | 2015-03-06 14:14:17.660768+09 | Backup method: pg_basebackup; --force: N
2 | standby_register | t | 2015-03-06 14:14:18.762222+09 |
4 | witness_create | t | 2015-03-06 14:14:22.072815+09 |
3 | standby_clone | t | 2015-03-06 14:14:23.524673+09 | Backup method: pg_basebackup; --force: N
3 | standby_register | t | 2015-03-06 14:14:24.620161+09 |
2 | repmgrd_start | t | 2015-03-06 14:14:29.639096+09 |
3 | repmgrd_start | t | 2015-03-06 14:14:29.641489+09 |
4 | repmgrd_start | t | 2015-03-06 14:14:29.648002+09 |
2 | standby_promote | t | 2015-03-06 14:15:01.956737+09 | Node 2 was successfully be promoted to master
2 | repmgrd_failover_promote | t | 2015-03-06 14:15:01.964771+09 | Node 2 promoted to master; old master 1 marked as failed
3 | repmgrd_failover_follow | t | 2015-03-06 14:15:07.228493+09 | Node 3 now following new upstream node 2
(12 rows)
This involves mainly abstracting the functions which copy
and create records from repmgr.c to dbutils.c, as they need
to be shared between repmgr and repmgrd.
Per issue noted here:
https://groups.google.com/forum/#!topic/repmgr/v5nu1Xwf6X0
Sometimes it's desirable to re-sync a "stale" data directory
on a standby, rather than start from scratch with pg_basebackup().
This re-adds the rsync code from the 2.x series, with some
modifications.
TODO: tablespace support.
Attempt to attach to the next available upstream node, otherwise
quit monitoring. We'll need to add further options for failover
scenarios, including attempting to attach to another node,
shutting down the server completely etc.
To handle cascaded replication we're going to have to keep track
of each node's upstream node. Also enumerate the node type
("primary", "standby" or "witness") and mark if active.
