By Mouhamadou Diaw
Barman is a tool to perform backup and recovery for PostgreSQL databases. It can do backup using two methods:
-rsync/ssh
-streaming
In this blog I am going to explain how to use these two methods to backup a PostgreSQL database. Generally it is a good practice to dedicate a server for barman instead of installing it on the database server. My environment is described below
postgreSQL server: dbi-pg-essentials 192.168.22.101 Centos 7
barman server: pgservertools 192.168.22.104 Oracle Linux 7
postgreSQL version: 11.1
barman version: 2.6
The first step is to install barman on the barman server pgservertools
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
[root@pgservertools network-scripts] # yum install barman.noarch barman-cli.noarch Loaded plugins: langpacks, ulninfo Resolving Dependencies --> Running transaction check ---> Package barman.noarch 0:2.6-1.rhel7 will be installed ---> Package barman-cli.noarch 0:1.3-1.rhel7.1 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: barman noarch 2.6-1.rhel7 pgdg10-updates-testing 300 k barman-cli noarch 1.3-1.rhel7.1 pgdg10-updates-testing 14 k Transaction Summary ================================================================================ ... ... Installed: barman.noarch 0:2.6-1.rhel7 barman-cli.noarch 0:1.3-1.rhel7.1 Complete! [root@pgservertools network-scripts] # |
The installation will create a linux user named barman.
As the rsync method need connections without passwords between two servers for the barman user, we have to configure ssh keys
On the server pgservertools (barman server) let’s create keys with the user barman and then copy the public key to the database server dbi-pg-essentials for the user postgres
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
|
- bash -4.2$ ssh -keygen Generating public /private rsa key pair. Enter file in which to save the key ( /var/lib/barman/ . ssh /id_rsa ): Created directory '/var/lib/barman/.ssh' . Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /var/lib/barman/ . ssh /id_rsa . Your public key has been saved in /var/lib/barman/ . ssh /id_rsa .pub. The key fingerprint is: f4:b7:6b:6e:38:25:ae:be:7f:9a:34:03:a1:1c:a0:ac barman@pgservertools The key's randomart image is: +--[ RSA 2048]----+ | . | | . . . | | o . o | | . . + o | |E o S . . | | ..... | | .++. | | .+++. | | .+++Bo | +-----------------+ - bash -4.2$ - bash -4.2$ cd . ssh / - bash -4.2$ ls id_rsa id_rsa.pub - bash -4.2$ ssh -copy- id postgres@dbi-pg-essentials The authenticity of host 'dbi-pg-essentials (192.168.22.101)' can't be established. ECDSA key fingerprint is 33:65:38:f4:eb:5b:f4:10:d3:36:7b:ea:5a:70:33:18. Are you sure you want to continue connecting ( yes /no )? yes /bin/ssh-copy-id : INFO: attempting to log in with the new key(s), to filter out any that are already installed /bin/ssh-copy-id : INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys postgres@dbi-pg-essentials's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'postgres@dbi-pg-essentials'" and check to make sure that only the key(s) you wanted were added. |
If everything is fine, barman should be able to connect to database server as postgres linux user without password
1
2
3
4
5
6
7
|
- bash -4.2$ hostname pgservertools.localdomain - bash -4.2$ id uid=994(barman) gid=992(barman) groups =992(barman) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 - bash -4.2$ ssh postgres@dbi-pg-essentials hostname dbi-pg-essentials - bash -4.2$ |
On the database server I also have installed the package barman-cli.noarch which will allow us to use the command barman-wal-archive. We will talk about this later.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
[root@dbi-pg-essentials ~] # yum install barman-cli.noarch Loaded plugins: fastestmirror dbipgessentials | 3.6 kB 00:00 edb-repos | 2.4 kB 00:00 … … Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : barman-cli-1.3-1.rhel7.1.noarch 1 /1 Verifying : barman-cli-1.3-1.rhel7.1.noarch 1 /1 Installed: barman-cli.noarch 0:1.3-1.rhel7.1 Complete! [root@dbi-pg-essentials ~] # |
rsync backup
As specified earlier in this case the backup is done using rsync. But we have many ways to ship WAL to the barman server. So before talking about barman configuration let’s take a moment to see the WAL shipping
1- With WAL archiving
To better understand I put this picture I got from the barman documentation . As we see backup is done via rsync and the WAL are sent via the barman-wal-archive. This utility comes with barman 2.6.
Before barman 2.6 the rsync command was used to send WAL to barman.
In the documentation we can find that using barman-wal-archive instead of rsync/SSH reduces the risk of data corruption of the shipped WAL file on the Barman server.
The use of barman_wal_archive is done via the postgresql.conf file on the database server. It appears on the value of the parameter archive_command. Below values in my postgresql.conf file.
1
2
3
4
|
[postgres@dbi-pg-essentials PG1]$ grep -E "archive_mode|wal_level|archive_command" postgresql.conf wal_level = replica # minimal, replica, or logical archive_mode = on # enables archiving; off, on, or always archive_command = 'barman-wal-archive 192.168.22.104 pgserver11 %p' # command to use to archive a logfile segment |
Before barman 2.6 we would use following for archive_command to send WAL to barman server
1
2
|
archive_command = 'rsync -a %p barman@pgservertools:/var/lib/barman/pgserver11/incoming/%f' # command to use to archive a logfile segment [postgres@dbi-pg-essentials PG1]$ |
2- With WAL archiving and WAL streaming
This picture from barman documentation will help to better understand
To use WAL streaming to the barman server, we need pg_receivewal (pg_receivexlog up to PostgreSQL 10) to be installed on the barman server. Be careful of the version of pg_receivewal. In my case I installed the version 11.1 as my PostgreSQL is 11.1
1
2
3
|
[postgres@pgservertools bin]$ /usr/pgsql-11/bin/pg_receivewal -V pg_receivewal (PostgreSQL) 11.1 [postgres@pgservertools bin]$ |
A streaming connection also should be configured and the parameter streaming_archiver should be set to on.
Now to resume let’s say that I want to configure barman with
-rsync method
-using barman_wal_archive and WAL streaming
The barman file configuration /etc/barman.conf should be like
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
[postgres@pgservertools bin]$ cat /etc/barman .conf | grep - v ^\; [barman] barman_user = barman configuration_files_directory = /etc/barman .d barman_home = /var/lib/barman log_file = /var/log/barman/barman .log log_level = INFO compression = gzip retention_policy = REDUNDANCY 2 [pgserver11] description = "Main DB Server" ssh_command = ssh postgres@dbi-pg-essentials streaming_conninfo=host=192.168.22.101 user=postgres conninfo=host=192.168.22.101 user=postgres backup_method = rsync streaming_archiver = on archiver = on path_prefix= /usr/pgsql-11/bin/ [postgres@pgservertools bin]$ |
And the postgresql.conf should contain following entries
1
2
3
4
|
[postgres@dbi-pg-essentials PG1]$ grep -E "archive_mode|wal_level|archive_command" postgresql.conf wal_level = replica # minimal, replica, or logical archive_mode = on # enables archiving; off, on, or always archive_command = 'barman-wal-archive 192.168.22.104 pgserver11 %p' # command to use to archive a logfile segment |
The first thing is to test that the barman configuration is fine for the PostgreSQL database. The check command should not return any errors. On the barman server with the user barman
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
|
- bash -4.2$ barman check pgserver11 Server pgserver11: PostgreSQL: OK is_superuser: OK PostgreSQL streaming: OK wal_level: OK directories: OK retention policy settings: OK backup maximum age: OK (no last_backup_maximum_age provided) compression settings: OK failed backups: OK (there are 0 failed backups) minimum redundancy requirements: OK (have 0 backups, expected at least 0) ssh : OK (PostgreSQL server) not in recovery: OK archive_mode: OK archive_command: OK continuous archiving: OK pg_receivexlog: OK pg_receivexlog compatible: OK receive-wal running: OK archiver errors: OK - bash -4.2$ |
Now we can launch a backup using the backup command on the barman server with the user barman
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
- bash -4.2$ barman backup pgserver11 Starting backup using rsync -exclusive method for server pgserver11 in /var/lib/barman/pgserver11/base/20190215T153350 Backup start at LSN: 0 /2E000060 (00000005000000000000002E, 00000060) This is the first backup for server pgserver11 Starting backup copy via rsync /SSH for 20190215T153350 Copy done ( time : 12 seconds) This is the first backup for server pgserver11 WAL segments preceding the current backup have been found: 00000005000000000000002D from server pgserver11 has been removed Asking PostgreSQL server to finalize the backup. Backup size: 74.1 MiB Backup end at LSN: 0 /2E000168 (00000005000000000000002E, 00000168) Backup completed (start time : 2019-02-15 15:33:52.392144, elapsed time : 15 seconds) Processing xlog segments from file archival for pgserver11 00000005000000000000002E 00000005000000000000002E.00000060.backup - bash -4.2$ |
We can list the existing backup. On the barman server with the barman user
1
2
3
|
- bash -4.2$ barman list-backup pgserver11 pgserver11 20190215T153350 - Fri Feb 15 15:34:08 2019 - Size: 74.1 MiB - WAL Size: 0 B - bash -4.2$ |
Streaming backup
Since the version 2.0, barman supports streaming replication for backup. This method uses the native pg_basebackup
1- Streaming-only backup
This picture is from the barman documentation may help
As we can see, In this case backup are done via streaming. WAL are also using streaming protocol.
2- WAL archiving and WAL streaming
Once again following picture may help
In this case we configure standard archiving as well to implement a more robust architecture
For example to implement a barman configuration with streaming backup WAL streaming and WAL archiving, the /etc/barman.conf should be like
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
[postgres@pgservertools bin]$ cat /etc/barman .conf | grep - v ^\; [barman] barman_user = barman configuration_files_directory = /etc/barman .d barman_home = /var/lib/barman log_file = /var/log/barman/barman .log log_level = INFO compression = gzip retention_policy = REDUNDANCY 2 [pgserver11] description = "Main DB Server" ssh_command = ssh postgres@dbi-pg-essentials streaming_conninfo=host=192.168.22.101 user=postgres conninfo=host=192.168.22.101 user=postgres backup_method = postgres streaming_archiver = on archiver = on slot_name=barman path_prefix= /usr/pgsql-11/bin/ |
and the postgressql.conf
1
2
3
4
|
[postgres@dbi-pg-essentials PG1]$ grep -E "archive_mode|wal_level|archive_command" postgresql.conf wal_level = replica # minimal, replica, or logical archive_mode = on # enables archiving; off, on, or always archive_command = 'barman-wal-archive 192.168.22.104 pgserver11 %p' # command to use to archive a logfile segment |
So the check should not return any errors
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
|
- bash -4.2$ barman check pgserver11 Server pgserver11: PostgreSQL: OK is_superuser: OK PostgreSQL streaming: OK wal_level: OK replication slot: OK directories: OK retention policy settings: OK backup maximum age: OK (no last_backup_maximum_age provided) compression settings: OK failed backups: OK (there are 0 failed backups) minimum redundancy requirements: OK (have 1 backups, expected at least 0) pg_basebackup: OK pg_basebackup compatible: OK pg_basebackup supports tablespaces mapping: OK archive_mode: OK archive_command: OK continuous archiving: OK pg_receivexlog: OK pg_receivexlog compatible: OK receive-wal running: OK archiver errors: OK - bash -4.2$ |
And we launch a backup, we can see that barman is using pg_basebackup
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
- bash -4.2$ barman backup pgserver11 Starting backup using postgres method for server pgserver11 in /var/lib/barman/pgserver11/base/20190215T160757 Backup start at LSN: 0 /2F0001A8 (00000005000000000000002F, 000001A8) Starting backup copy via pg_basebackup for 20190215T160757 Copy done ( time : 11 seconds) Finalising the backup. Backup size: 73.9 MiB Backup end at LSN: 0 /31000060 (000000050000000000000031, 00000060) Backup completed (start time : 2019-02-15 16:07:57.919595, elapsed time : 11 seconds) Processing xlog segments from file archival for pgserver11 000000050000000000000030 000000050000000000000031 Processing xlog segments from streaming for pgserver11 000000050000000000000030 - bash -4.2$ |
Available backups are now
1
2
3
|
- bash -4.2$ barman list-backup pgserver11 pgserver11 20190215T160757 - Fri Feb 15 16:08:09 2019 - Size: 73.9 MiB - WAL Size: 0 B pgserver11 20190215T153350 - Fri Feb 15 15:34:08 2019 - Size: 74.1 MiB - WAL Size: 48.3 KiB |
To restore with barman, we use the command recover. For example the following command will restore the backup 20190215T160757 on server dbi-pg-essentials_3 in the directory /u02/pgdata/PGRESTORE
1
|
- bash -4.2$ barman recover --remote- ssh - command "ssh postgres@dbi-pg-essentials_3" pgserver11 20190215T160757 /u02/pgdata/PGRESTORE |
Conclusion
In this blog I have tried to explain different scenarios for using barman. We talked about rsync method and streaming methods for backups. Before starting the setup a choice must be done. One can check documentation for more information
Ref: http://docs.pgbarman.org/