Infrastructure at your Service

Daniel Westermann

What are these *.ready and *.done files for in PostgreSQL?

When you run PostgreSQL workloads in production you must have a backup and restore implementation. Even for development instances, which are like production for the developers, a well-tested backup and restore procedure sometimes must be in place. Community PostgreSQL comes with pg_basebackup to help you with creating a full consistent backup of your PostgreSQL cluster. If you want to be able to do point in time recovery (PITR) you need to archive the WAL segments which can later be used to roll forward from a base backup. WAL segments go to the pg_wal directory, but if you have closer look into this directory you will see more than just the WAL segments in case you have enabled archiving.

IF you look at a freshly initialized PostgreSQL cluster the content of pg_wal looks like this:

[email protected]:/home/postgres/ [pgdev] ls -la $PGDATA/pg_wal
total 16396
drwx------  3 postgres postgres     4096 Jul  4 11:47 .
drwx------ 20 postgres postgres     4096 Jul  4 11:47 ..
-rw-------  1 postgres postgres 16777216 Jul  4 11:52 000000010000000000000001
drwx------  2 postgres postgres     4096 Jul  4 11:47 archive_status

There is one WAL segment and a directory called “archived_status” which is empty right now:

[email protected]:/home/postgres/ [pgdev] ls -la $PGDATA/pg_wal/archive_status/
total 8
drwx------ 2 postgres postgres 4096 Jul  4 11:47 .
drwx------ 3 postgres postgres 4096 Jul  4 11:47 ..

To see what’s going on in this directory we need to enable archiving and tell PostgreSQL what do when a WAL segment needs to be archived:

[email protected]:/home/postgres/ [pgdev] mkdir /tmp/archived_wal
[email protected]:/home/postgres/ [pgdev] psql -c "alter system set archive_mode='on'" postgres
ALTER SYSTEM
[email protected]:/home/postgres/ [pgdev] psql -c "alter system set archive_command='test ! -f /tmp/archived_wal/%f && cp %p /tmp/archived_wal/%f'" postgres
ALTER SYSTEM

Of course you should not archive to /tmp and you also should not use cp, as this does not guarantee that the archived segment really is written to disk. For the scope of this post it is fine like that.

Changing the archive command can be done online, but enabling or disabling archiving requires a restart:

[email protected]:/home/postgres/ [pgdev] pg_ctl restart

Let’s see what happens if we force the current segment to be archived:

[email protected]:/home/postgres/ [pgdev] psql -c "select pg_switch_wal()" postgres
 pg_switch_wal 
---------------
 0/167AF68
(1 row)
[email protected]:/home/postgres/ [pgdev] ls -la /tmp/archived_wal/
total 16392
drwxr-xr-x  2 postgres postgres     4096 Jul  4 12:08 .
drwxrwxrwt 10 root     root         4096 Jul  4 12:07 ..
-rw-------  1 postgres postgres 16777216 Jul  4 12:08 000000010000000000000001

As expected we see the archived segment in the directory we specified in the archive command. In addition PostgreSQL generated a “.done” file in the archive_status directory in pg_wal:

[email protected]:/home/postgres/ [pgdev] ls -la $PGDATA/pg_wal/archive_status/
total 8
drwx------ 2 postgres postgres 4096 Jul  4 12:08 .
drwx------ 3 postgres postgres 4096 Jul  4 12:08 ..
-rw------- 1 postgres postgres    0 Jul  4 12:08 000000010000000000000001.done

So, whenever a segment was successfully archived you get the corresponding “.done” file. When do you see “.ready” files then? The answer is simple: Exactly when the opposite happens: Archiving of a segment failed for whatever reason:

[email protected]:/home/postgres/ [pgdev] psql -c "alter system set archive_command='/bin/false'" postgres
ALTER SYSTEM
[email protected]:/home/postgres/ [pgdev] psql -c "select pg_reload_conf()";
 pg_reload_conf 
----------------
 t
(1 row)
[email protected]:/home/postgres/ [pgdev] psql -c "select pg_switch_wal()" postgres
 pg_switch_wal 
---------------
 0/2000160
(1 row)
[email protected]:/home/postgres/ [pgdev] ls -la $PGDATA/pg_wal/archive_status/
total 8
drwx------ 2 postgres postgres 4096 Jul  4 12:13 .
drwx------ 3 postgres postgres 4096 Jul  4 12:12 ..
-rw------- 1 postgres postgres    0 Jul  4 12:13 000000010000000000000002.ready

This file will be there until archiving succeeds again:

[email protected]:/home/postgres/ [pgdev] psql -c "alter system set archive_command='test ! -f /tmp/archived_wal/%f && cp %p /tmp/archived_wal/%f'" postgres
ALTER SYSTEM
[email protected]:/home/postgres/ [pgdev] psql -c "select pg_reload_conf()";
 pg_reload_conf 
----------------
 t
(1 row)
[email protected]:/home/postgres/ [pgdev] ls -la $PGDATA/pg_wal/archive_status/
total 8
drwx------ 2 postgres postgres 4096 Jul  4 12:15 .
drwx------ 3 postgres postgres 4096 Jul  4 12:12 ..
-rw------- 1 postgres postgres    0 Jul  4 12:13 000000010000000000000002.done

When you see many “.ready” files this can either mean your archive command is currently failing or load on the system is so high, that archiving can not keep up.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Daniel Westermann
Daniel Westermann

Principal Consultant & Technology Leader Open Infrastructure