Infrastructure at your Service

Daniel Westermann

Minor PostgreSQL version upgrades in a Patroni cluster

Last week PostgreSQL 14.1, and all other minor versions for the supported version of PostgreSQL have been released. As usual, it is highly recommended to update to the latest minor release as soon as possible. Minor version updates of PostgreSQL are just a matter of installing the new binaries, shutting down the old cluster and starting it up again using the new binaries, that’s it. If you are using packages the procedure is to shutdown the cluster, install the new packages and start it up again. Nothing really complicated in either case and really easy to do. What we’ll look into in this post is, how you can do minor version PostgreSQL updates if you run Patroni on top of PostgreSQL for implementing a high available solution.

My current Patroni configuration looks like this:

[email protected]:/home/postgres/ [PG1] patronictl list
+----------+-----------------+---------+---------+----+-----------+
| Member   | Host            | Role    | State   | TL | Lag in MB |
+ Cluster: postgres_ha (7030448849176438024) ----+----+-----------+
| patroni1 | 192.168.100.170 | Leader  | running |  4 |           |
| patroni2 | 192.168.100.171 | Replica | running |  4 |         0 |
| patroni3 | 192.168.100.172 | Replica | running |  4 |         0 |
+----------+-----------------+---------+---------+----+-----------+

There is a primary/leader instance which feeds to replicas, all of them running PostgreSQL 14.0:

[email protected]:/home/postgres/ [PG1] psql -c "select version()";
                                              version                                              
---------------------------------------------------------------------------------------------------
 PostgreSQL 14.0 on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
(1 row)

What do we need to do to update this configuration to PostgreSQL 14.1? I’ve installed from source code, so the first step is to install the new version of PostgreSQL. This is already done in may case:

[email protected]:/home/postgres/ [PG1] ls -l /u01/app/postgres/product/14/
total 8
drwxr-xr-x 6 postgres postgres 4096 Nov 10 20:43 db_0
drwxr-xr-x 6 postgres postgres 4096 Nov 11 15:35 db_1

“db_0” is PostgreSQL 14.0 and “db_1” is PostgreSQL 14.1. This is our standard, in your case this of course very probably looks different. Having both versions installed on all nodes already, what is the next step? First you need think about which node you want to upgrade first. We usually go for one of the replicas and in the case of PostgreSQL 14.1 this is even recommended as one patch (“Fix physical replication for cases where the primary crashes after shipping a WAL segment that ends with a partial WAL record (Álvaro Herrera)”) should be applied on the replica first.

Let’s start with the third node (patroni3/192.168.100.172). First we need to stop Patroni on that host:

[email protected]:/home/postgres/ [PG1] sudo systemctl stop patroni

Checking the Patroni status afterwards, we can see that the node is now gone:

[email protected]:/home/postgres/ [PG1] patronictl list
+----------+-----------------+---------+---------+----+-----------+
| Member   | Host            | Role    | State   | TL | Lag in MB |
+ Cluster: postgres_ha (7030448849176438024) ----+----+-----------+
| patroni1 | 192.168.100.170 | Leader  | running |  4 |           |
| patroni2 | 192.168.100.171 | Replica | running |  4 |         0 |
+----------+-----------------+---------+---------+----+-----------+

All we need to change is the path to the PostgreSQL binaries in the Patroni configuration file. Current I have this:

[email protected]:/home/postgres/ [PG1] grep bin_dir /u01/app/postgres/local/dmk/etc/patroni.yml
  bin_dir: /u01/app/postgres/product/14/db_0/bin

… which points to my PostgreSQL 14.0 installation. After changing that:

[email protected]:/home/postgres/ [PG1] sed -i 's#14/db_0#14/db_1#g' /u01/app/postgres/local/dmk/etc/patroni.yml
[email protected]:/home/postgres/ [PG1] grep bin_dir /u01/app/postgres/local/dmk/etc/patroni.yml
  bin_dir: /u01/app/postgres/product/14/db_1/bin

Once this is done, start the Patroni service and this replica is patched:

[email protected]:/home/postgres/ [PG1] sudo systemctl start patroni
[email protected]:/home/postgres/ [PG1] psql -c "select version()"
                                              version                                              
---------------------------------------------------------------------------------------------------
 PostgreSQL 14.1 on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
(1 row)

Checking the status of the Patroni configuration the replica is back:

[email protected]:/home/postgres/ [PG1] patronictl list
+----------+-----------------+---------+---------+----+-----------+
| Member   | Host            | Role    | State   | TL | Lag in MB |
+ Cluster: postgres_ha (7030448849176438024) ----+----+-----------+
| patroni1 | 192.168.100.170 | Leader  | running |  4 |           |
| patroni2 | 192.168.100.171 | Replica | running |  4 |         0 |
| patroni3 | 192.168.100.172 | Replica | running |  4 |         0 |
+----------+-----------------+---------+---------+----+-----------+

That was quite easy and you can follow the exact same steps for the second replica. If you prepare that well you’ll have a downtime of a few seconds per replica. The primary, as of now, did not have any downtime. This starts now and you have two choices: If you want that the primary stays on the same host, then you can disable auto failover in Patroni, patch the primary, bring it up again and finally enable auto failover. Another option is to switch the primary to another node, so the current primary becomes a new replica. Once roles are switched you can follow the procedure outlined above.

We’ll do the switchover now:

[email protected]:/home/postgres/ [PG1] patronictl switchover
Master [patroni1]: patroni1
Candidate ['patroni2', 'patroni3'] []: patroni2
When should the switchover take place (e.g. 2021-11-14T19:45 )  [now]: now
Current cluster topology
+----------+-----------------+---------+---------+----+-----------+
| Member   | Host            | Role    | State   | TL | Lag in MB |
+ Cluster: postgres_ha (7030448849176438024) ----+----+-----------+
| patroni1 | 192.168.100.170 | Leader  | running |  4 |           |
| patroni2 | 192.168.100.171 | Replica | running |  4 |         0 |
| patroni3 | 192.168.100.172 | Replica | running |  4 |         0 |
+----------+-----------------+---------+---------+----+-----------+
Are you sure you want to switchover cluster postgres_ha, demoting current master patroni1? [y/N]: y

2021-11-14 18:45:29.79070 Successfully switched over to "patroni2"
+----------+-----------------+---------+---------+----+-----------+
| Member   | Host            | Role    | State   | TL | Lag in MB |
+ Cluster: postgres_ha (7030448849176438024) ----+----+-----------+
| patroni1 | 192.168.100.170 | Replica | stopped |    |   unknown |
| patroni2 | 192.168.100.171 | Leader  | running |  4 |           |
| patroni3 | 192.168.100.172 | Replica | running |  4 |         0 |
+----------+-----------------+---------+---------+----+-----------+

[email protected]:/home/postgres/ [PG1] patronictl list
+----------+-----------------+---------+---------+----+-----------+
| Member   | Host            | Role    | State   | TL | Lag in MB |
+ Cluster: postgres_ha (7030448849176438024) ----+----+-----------+
| patroni1 | 192.168.100.170 | Replica | running |  5 |         0 |
| patroni2 | 192.168.100.171 | Leader  | running |  5 |           |
| patroni3 | 192.168.100.172 | Replica | running |  5 |         0 |
+----------+-----------------+---------+---------+----+-----------+

Now we can follow the same procedure on the old primary:

[email protected]:/home/postgres/ [PG1] hostname
patroni1
[email protected]:/home/postgres/ [PG1] sudo systemctl stop patroni
[email protected]:/home/postgres/ [PG1] sed -i 's#14/db_0#14/db_1#g' /u01/app/postgres/local/dmk/etc/patroni.yml
[email protected]:/home/postgres/ [PG1] grep bin_dir /u01/app/postgres/local/dmk/etc/patroni.yml
  bin_dir: /u01/app/postgres/product/14/db_1/bin
[email protected]:/home/postgres/ [PG1] sudo systemctl start patroni
[email protected]:/home/postgres/ [PG1] patronictl list
+----------+-----------------+---------+---------+----+-----------+
| Member   | Host            | Role    | State   | TL | Lag in MB |
+ Cluster: postgres_ha (7030448849176438024) ----+----+-----------+
| patroni1 | 192.168.100.170 | Replica | running |  5 |         0 |
| patroni2 | 192.168.100.171 | Leader  | running |  5 |           |
| patroni3 | 192.168.100.172 | Replica | running |  5 |         0 |
+----------+-----------------+---------+---------+----+-----------+
[email protected]:/home/postgres/ [PG1] psql -c "select version()";
                                              version                                              
---------------------------------------------------------------------------------------------------
 PostgreSQL 14.1 on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
(1 row)

All done.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Daniel Westermann
Daniel Westermann

Principal Consultant & Technology Leader Open Infrastructure