Infrastructure at your Service

ZFS for Solaris is around for several years now (since 2015). But there is also a project called OpenZFS which makes ZFS available on other operating systems. For Linux the announcement for ZFS being production ready was back in 2013. So why not run PostgreSQL on it? ZFS provides many cool features including compression, snapshots and build in volume management. Lets give it a try and do an initial setup. More details will follow in separate posts.

As usual I am running a CentOS 7 VM for my tests:

[[email protected] ~] lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 7.2.1511 (Core) 
Release:	7.2.1511
Codename:	Core

There is a dedicated website for ZFS on Linux where you can find the instructions on how to install it for various distributions. The instruction for CentOS/RHEL are quite easy. Download the repo files:

[[email protected] ~] yum install http://download.zfsonlinux.org/epel/zfs-release$(rpm -E %dist).noarch.rpm
Loaded plugins: fastestmirror
zfs-release.el7.centos.noarch.rpm                                                                    | 5.0 kB  00:00:00     
Examining /var/tmp/yum-root-Uv79vc/zfs-release.el7.centos.noarch.rpm: zfs-release-1-3.el7.centos.noarch
Marking /var/tmp/yum-root-Uv79vc/zfs-release.el7.centos.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package zfs-release.noarch 0:1-3.el7.centos will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================
 Package                  Arch                Version                     Repository                                   Size
============================================================================================================================
Installing:
 zfs-release              noarch              1-3.el7.centos              /zfs-release.el7.centos.noarch              2.9 k

Transaction Summary
============================================================================================================================
Install  1 Package

Total size: 2.9 k
Installed size: 2.9 k
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : zfs-release-1-3.el7.centos.noarch                                                                        1/1 
  Verifying  : zfs-release-1-3.el7.centos.noarch                                                                        1/1 

Installed:
  zfs-release.noarch 0:1-3.el7.centos                                                                                       

Complete!

[[email protected] ~] gpg --quiet --with-fingerprint /etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
gpg: new configuration file `/root/.gnupg/gpg.conf' created
gpg: WARNING: options in `/root/.gnupg/gpg.conf' are not yet active during this run
pub  2048R/F14AB620 2013-03-21 ZFS on Linux 
      Key fingerprint = C93A FFFD 9F3F 7B03 C310  CEB6 A9D5 A1C0 F14A B620
sub  2048R/99685629 2013-03-21

For the next step it depends if you want to go with DKMS or kABI-tracking kmod. I’ll go with kABI-tracking kmod and therefore will disable the DKMS repository and enable the kmod repository:

[[email protected] ~] cat /etc/yum.repos.d/zfs.repo 
[zfs]
name=ZFS on Linux for EL7 - dkms
baseurl=http://download.zfsonlinux.org/epel/7/$basearch/
enabled=0
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux

[zfs-kmod]
name=ZFS on Linux for EL7 - kmod
baseurl=http://download.zfsonlinux.org/epel/7/kmod/$basearch/
enabled=1
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux

[zfs-source]
name=ZFS on Linux for EL7 - Source
baseurl=http://download.zfsonlinux.org/epel/7/SRPMS/
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux

[zfs-testing]
name=ZFS on Linux for EL7 - dkms - Testing
baseurl=http://download.zfsonlinux.org/epel-testing/7/$basearch/
enabled=0
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux

[zfs-testing-kmod]
name=ZFS on Linux for EL7 - kmod - Testing
baseurl=http://download.zfsonlinux.org/epel-testing/7/kmod/$basearch/
enabled=0
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux

[zfs-testing-source]
name=ZFS on Linux for EL7 - Testing Source
baseurl=http://download.zfsonlinux.org/epel-testing/7/SRPMS/
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
[[email protected] ~] 

Installing ZFS from here on is just a matter of using yum:

[[email protected] ~] yum install zfs
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.spreitzer.ch
 * extras: mirror.spreitzer.ch
 * updates: mirror.de.leaseweb.net
zfs-kmod/x86_64/primary_db                                                                           | 231 kB  00:00:01     
Resolving Dependencies
--> Running transaction check
---> Package zfs.x86_64 0:0.6.5.8-1.el7.centos will be installed
--> Processing Dependency: zfs-kmod = 0.6.5.8 for package: zfs-0.6.5.8-1.el7.centos.x86_64
--> Processing Dependency: spl = 0.6.5.8 for package: zfs-0.6.5.8-1.el7.centos.x86_64
--> Processing Dependency: libzpool2 = 0.6.5.8 for package: zfs-0.6.5.8-1.el7.centos.x86_64
--> Processing Dependency: libzfs2 = 0.6.5.8 for package: zfs-0.6.5.8-1.el7.centos.x86_64
--> Processing Dependency: libuutil1 = 0.6.5.8 for package: zfs-0.6.5.8-1.el7.centos.x86_64
--> Processing Dependency: libnvpair1 = 0.6.5.8 for package: zfs-0.6.5.8-1.el7.centos.x86_64
--> Processing Dependency: libzpool.so.2()(64bit) for package: zfs-0.6.5.8-1.el7.centos.x86_64
--> Processing Dependency: libzfs_core.so.1()(64bit) for package: zfs-0.6.5.8-1.el7.centos.x86_64
--> Processing Dependency: libzfs.so.2()(64bit) for package: zfs-0.6.5.8-1.el7.centos.x86_64
--> Processing Dependency: libuutil.so.1()(64bit) for package: zfs-0.6.5.8-1.el7.centos.x86_64
--> Processing Dependency: libnvpair.so.1()(64bit) for package: zfs-0.6.5.8-1.el7.centos.x86_64
--> Running transaction check
---> Package kmod-zfs.x86_64 0:0.6.5.8-1.el7.centos will be installed
--> Processing Dependency: spl-kmod for package: kmod-zfs-0.6.5.8-1.el7.centos.x86_64
---> Package libnvpair1.x86_64 0:0.6.5.8-1.el7.centos will be installed
---> Package libuutil1.x86_64 0:0.6.5.8-1.el7.centos will be installed
---> Package libzfs2.x86_64 0:0.6.5.8-1.el7.centos will be installed
---> Package libzpool2.x86_64 0:0.6.5.8-1.el7.centos will be installed
---> Package spl.x86_64 0:0.6.5.8-1.el7.centos will be installed
--> Running transaction check
---> Package kmod-spl.x86_64 0:0.6.5.8-1.el7.centos will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================
 Package                     Arch                    Version                                Repository                 Size
============================================================================================================================
Installing:
 zfs                         x86_64                  0.6.5.8-1.el7.centos                   zfs-kmod                  334 k
Installing for dependencies:
 kmod-spl                    x86_64                  0.6.5.8-1.el7.centos                   zfs-kmod                  110 k
 kmod-zfs                    x86_64                  0.6.5.8-1.el7.centos                   zfs-kmod                  665 k
 libnvpair1                  x86_64                  0.6.5.8-1.el7.centos                   zfs-kmod                   35 k
 libuutil1                   x86_64                  0.6.5.8-1.el7.centos                   zfs-kmod                   41 k
 libzfs2                     x86_64                  0.6.5.8-1.el7.centos                   zfs-kmod                  123 k
 libzpool2                   x86_64                  0.6.5.8-1.el7.centos                   zfs-kmod                  423 k
 spl                         x86_64                  0.6.5.8-1.el7.centos                   zfs-kmod                   29 k

Transaction Summary
============================================================================================================================
Install  1 Package (+7 Dependent packages)

Total download size: 1.7 M
Installed size: 5.7 M
Is this ok [y/d/N]: y
Downloading packages:
(1/8): kmod-spl-0.6.5.8-1.el7.centos.x86_64.rpm                                                      | 110 kB  00:00:01     
(2/8): libnvpair1-0.6.5.8-1.el7.centos.x86_64.rpm                                                    |  35 kB  00:00:00     
(3/8): libuutil1-0.6.5.8-1.el7.centos.x86_64.rpm                                                     |  41 kB  00:00:00     
(4/8): kmod-zfs-0.6.5.8-1.el7.centos.x86_64.rpm                                                      | 665 kB  00:00:02     
(5/8): libzfs2-0.6.5.8-1.el7.centos.x86_64.rpm                                                       | 123 kB  00:00:00     
(6/8): libzpool2-0.6.5.8-1.el7.centos.x86_64.rpm                                                     | 423 kB  00:00:00     
(7/8): spl-0.6.5.8-1.el7.centos.x86_64.rpm                                                           |  29 kB  00:00:00     
(8/8): zfs-0.6.5.8-1.el7.centos.x86_64.rpm                                                           | 334 kB  00:00:00     
----------------------------------------------------------------------------------------------------------------------------
Total                                                                                       513 kB/s | 1.7 MB  00:00:03     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : libuutil1-0.6.5.8-1.el7.centos.x86_64                                                                    1/8 
  Installing : libnvpair1-0.6.5.8-1.el7.centos.x86_64                                                                   2/8 
  Installing : libzpool2-0.6.5.8-1.el7.centos.x86_64                                                                    3/8 
  Installing : kmod-spl-0.6.5.8-1.el7.centos.x86_64                                                                     4/8 
  Installing : spl-0.6.5.8-1.el7.centos.x86_64                                                                          5/8 
  Installing : libzfs2-0.6.5.8-1.el7.centos.x86_64                                                                      6/8 
  Installing : kmod-zfs-0.6.5.8-1.el7.centos.x86_64                                                                     7/8 
  Installing : zfs-0.6.5.8-1.el7.centos.x86_64                                                                          8/8 
  Verifying  : libnvpair1-0.6.5.8-1.el7.centos.x86_64                                                                   1/8 
  Verifying  : libzfs2-0.6.5.8-1.el7.centos.x86_64                                                                      2/8 
  Verifying  : zfs-0.6.5.8-1.el7.centos.x86_64                                                                          3/8 
  Verifying  : spl-0.6.5.8-1.el7.centos.x86_64                                                                          4/8 
  Verifying  : kmod-zfs-0.6.5.8-1.el7.centos.x86_64                                                                     5/8 
  Verifying  : libzpool2-0.6.5.8-1.el7.centos.x86_64                                                                    6/8 
  Verifying  : libuutil1-0.6.5.8-1.el7.centos.x86_64                                                                    7/8 
  Verifying  : kmod-spl-0.6.5.8-1.el7.centos.x86_64                                                                     8/8 

Installed:
  zfs.x86_64 0:0.6.5.8-1.el7.centos                                                                                         

Dependency Installed:
  kmod-spl.x86_64 0:0.6.5.8-1.el7.centos   kmod-zfs.x86_64 0:0.6.5.8-1.el7.centos  libnvpair1.x86_64 0:0.6.5.8-1.el7.centos 
  libuutil1.x86_64 0:0.6.5.8-1.el7.centos  libzfs2.x86_64 0:0.6.5.8-1.el7.centos   libzpool2.x86_64 0:0.6.5.8-1.el7.centos  
  spl.x86_64 0:0.6.5.8-1.el7.centos       

Complete!
[[email protected] ~]

Be aware that the kernel modules are not loaded by default, so you have to do this on your own:

[[email protected] ~] /sbin/modprobe zfs
Last login: Wed Sep 28 11:04:21 2016 from 192.168.22.1
[[email protected] ~]$ lsmod | grep zfs
zfs                  2713912  0 
zunicode              331170  1 zfs
zavl                   15236  1 zfs
zcommon                55411  1 zfs
znvpair                93227  2 zfs,zcommon
spl                    92223  3 zfs,zcommon,znvpair
[[email protected] ~] zfs list
no datasets available

For loading the modules automatically create a file under /etc/modules-load.d:

[[email protected] ~] echo "zfs" > /etc/modules-load.d/zfs.conf
[[email protected] ~] cat /etc/modules-load.d/zfs.conf
zfs

So far so good. Lets create a ZFS file system. I have two disks available for playing with ZFS (sdb and sdc):

[[email protected] ~] ls -la /dev/sd*
brw-rw----. 1 root disk 8,  0 Sep 28 11:14 /dev/sda
brw-rw----. 1 root disk 8,  1 Sep 28 11:14 /dev/sda1
brw-rw----. 1 root disk 8,  2 Sep 28 11:14 /dev/sda2
brw-rw----. 1 root disk 8, 16 Sep 28 11:14 /dev/sdb
brw-rw----. 1 root disk 8, 32 Sep 28 11:14 /dev/sdc

The first thing you have to do is to create a new zfs pool (I don’t care about the warnings, that is why I use the “-f” option below):

[[email protected] ~] zpool create pgpool mirror /dev/sdb /dev/sdc
invalid vdev specification
use '-f' to override the following errors:
/dev/sdb does not contain an EFI label but it may contain partition information in the MBR.
/dev/sdc does not contain an EFI label but it may contain partition information in the MBR.
[[email protected] ~] zpool create pgpool mirror /dev/sdb /dev/sdc -f
[[email protected] ~] zpool list
NAME     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
pgpool  9.94G    65K  9.94G         -     0%     0%  1.00x  ONLINE  -
[[email protected] ~] zpool status pgpool
  pool: pgpool
 state: ONLINE
  scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	pgpool      ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    sdb     ONLINE       0     0     0
	    sdc     ONLINE       0     0     0

errors: No known data errors

[[email protected] ~] df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   49G  1.7G   47G   4% /
devtmpfs                 235M     0  235M   0% /dev
tmpfs                    245M     0  245M   0% /dev/shm
tmpfs                    245M  4.3M  241M   2% /run
tmpfs                    245M     0  245M   0% /sys/fs/cgroup
/dev/sda1                497M  291M  206M  59% /boot
tmpfs                     49M     0   49M   0% /run/user/1000
pgpool                   9.7G     0  9.7G   0% /pgpool

What I did here is to create a mirrored pool over my two disks. The open zfs wiki has some performance tips for running PostgreSQL on ZFS as well as for other topics. Lets go with the recommendations:

[[email protected] ~] zfs create pgpool/pgdata -o recordsize=8192
[[email protected] ~] zfs set logbias=throughput pgpool/pgdata
[[email protected] ~] zfs set primarycache=all pgpool/pgdata
[[email protected] ~] zfs list
NAME            USED  AVAIL  REFER  MOUNTPOINT
pgpool           82K  9.63G  19.5K  /pgpool
pgpool/pgdata    19K  9.63G    19K  /pgpool/pgdata

My new ZFS file system is ready and already mounted, cool. Lets change the permissions and list all the properties:

[[email protected] ~] chown postgres:postgres /pgpool/pgdata
[[email protected] ~] zfs get all /pgpool/pgdata
NAME           PROPERTY              VALUE                  SOURCE
pgpool/pgdata  type                  filesystem             -
pgpool/pgdata  creation              Wed Sep 28 11:31 2016  -
pgpool/pgdata  used                  19K                    -
pgpool/pgdata  available             9.63G                  -
pgpool/pgdata  referenced            19K                    -
pgpool/pgdata  compressratio         1.00x                  -
pgpool/pgdata  mounted               yes                    -
pgpool/pgdata  quota                 none                   default
pgpool/pgdata  reservation           none                   default
pgpool/pgdata  recordsize            8K                     local
pgpool/pgdata  mountpoint            /pgpool/pgdata         default
pgpool/pgdata  sharenfs              off                    default
pgpool/pgdata  checksum              on                     default
pgpool/pgdata  compression           off                    default
pgpool/pgdata  atime                 on                     default
pgpool/pgdata  devices               on                     default
pgpool/pgdata  exec                  on                     default
pgpool/pgdata  setuid                on                     default
pgpool/pgdata  readonly              off                    default
pgpool/pgdata  zoned                 off                    default
pgpool/pgdata  snapdir               hidden                 default
pgpool/pgdata  aclinherit            restricted             default
pgpool/pgdata  canmount              on                     default
pgpool/pgdata  xattr                 on                     default
pgpool/pgdata  copies                1                      default
pgpool/pgdata  version               5                      -
pgpool/pgdata  utf8only              off                    -
pgpool/pgdata  normalization         none                   -
pgpool/pgdata  casesensitivity       sensitive              -
pgpool/pgdata  vscan                 off                    default
pgpool/pgdata  nbmand                off                    default
pgpool/pgdata  sharesmb              off                    default
pgpool/pgdata  refquota              none                   default
pgpool/pgdata  refreservation        none                   default
pgpool/pgdata  primarycache          all                    default
pgpool/pgdata  secondarycache        all                    default
pgpool/pgdata  usedbysnapshots       0                      -
pgpool/pgdata  usedbydataset         19K                    -
pgpool/pgdata  usedbychildren        0                      -
pgpool/pgdata  usedbyrefreservation  0                      -
pgpool/pgdata  logbias               throughput             local
pgpool/pgdata  dedup                 off                    default
pgpool/pgdata  mlslabel              none                   default
pgpool/pgdata  sync                  standard               default
pgpool/pgdata  refcompressratio      1.00x                  -
pgpool/pgdata  written               19K                    -
pgpool/pgdata  logicalused           9.50K                  -
pgpool/pgdata  logicalreferenced     9.50K                  -
pgpool/pgdata  filesystem_limit      none                   default
pgpool/pgdata  snapshot_limit        none                   default
pgpool/pgdata  filesystem_count      none                   default
pgpool/pgdata  snapshot_count        none                   default
pgpool/pgdata  snapdev               hidden                 default
pgpool/pgdata  acltype               off                    default
pgpool/pgdata  context               none                   default
pgpool/pgdata  fscontext             none                   default
pgpool/pgdata  defcontext            none                   default
pgpool/pgdata  rootcontext           none                   default
pgpool/pgdata  relatime              on                     temporary
pgpool/pgdata  redundant_metadata    all                    default
pgpool/pgdata  overlay               off                    default

Ready to deploy a PostgreSQL instance on it:

[email protected]:/home/postgres/ [pg954] initdb -D /pgpool/pgdata/
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locales
  COLLATE:  en_US.UTF-8
  CTYPE:    en_US.UTF-8
  MESSAGES: en_US.UTF-8
  MONETARY: de_CH.UTF-8
  NUMERIC:  de_CH.UTF-8
  TIME:     en_US.UTF-8
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /pgpool/pgdata ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /pgpool/pgdata/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
syncing data to disk ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    pg_ctl -D /pgpool/pgdata/ -l logfile start

Startup:

[email protected]:/home/postgres/ [pg954] mkdir /pgpool/pgdata/pg_log
[email protected]:/home/postgres/ [pg954] sed -i 's/logging_collector = off/logging_collector = on/g' /pgpool/pgdata/postgresql.conf
[email protected]:/home/postgres/ [pg954] pg_ctl -D /pgpool/pgdata/ start
[email protected]centos7:/home/postgres/ [pg954] psql postgres
psql (9.5.4 dbi services build)
Type "help" for help.

postgres=

Ready. Lets reboot and check if the ZFS file system is mounted automatically:

[email protected]:/home/postgres/ [pg954] df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   49G  1.8G   47G   4% /
devtmpfs                 235M     0  235M   0% /dev
tmpfs                    245M     0  245M   0% /dev/shm
tmpfs                    245M  4.3M  241M   2% /run
tmpfs                    245M     0  245M   0% /sys/fs/cgroup
/dev/sda1                497M  291M  206M  59% /boot
tmpfs                     49M     0   49M   0% /run/user/1000
[email protected]:/home/postgres/ [pg954] lsmod | grep zfs
zfs                  2713912  0 
zunicode              331170  1 zfs
zavl                   15236  1 zfs
zcommon                55411  1 zfs
znvpair                93227  2 zfs,zcommon
spl                    92223  3 zfs,zcommon,znvpair

Gone. The kernel modules are loaded but the file system was not mounted. What to do?

[[email protected] ~] zpool list
no pools available
[[email protected] ~] zpool import pgpool
[[email protected] ~] zpool list
NAME     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
pgpool  9.94G  39.3M  9.90G         -     0%     0%  1.00x  ONLINE  -
[[email protected] ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   49G  1.8G   47G   4% /
devtmpfs                 235M     0  235M   0% /dev
tmpfs                    245M     0  245M   0% /dev/shm
tmpfs                    245M  4.3M  241M   2% /run
tmpfs                    245M     0  245M   0% /sys/fs/cgroup
/dev/sda1                497M  291M  206M  59% /boot
tmpfs                     49M     0   49M   0% /run/user/1000
pgpool                   9.6G     0  9.6G   0% /pgpool
pgpool/pgdata            9.7G   39M  9.6G   1% /pgpool/pgdata

Ok, how to auto mount?

[[email protected] ~] systemctl enable zfs-mount
[[email protected] ~] systemctl enable zfs-import-cache
[[email protected] ~] reboot

I am not sure why this is necessary, should happen automatically.

PS: There is an interesting discussion about PostgreSQL on ZFS on the PostgreSQL performance mailing list currently.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Daniel Westermann
Daniel Westermann

Principal Consultant & Technology Leader Open Infrastructure