ZFS for Solaris is around for several years now (since 2015). But there is also a project called OpenZFS which makes ZFS available on other operating systems. For Linux the announcement for ZFS being production ready was back in 2013. So why not run PostgreSQL on it? ZFS provides many cool features including compression, snapshots and build in volume management. Lets give it a try and do an initial setup. More details will follow in separate posts.
As usual I am running a CentOS 7 VM for my tests:
[root@centos7 ~] lsb_release -a LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch Distributor ID: CentOS Description: CentOS Linux release 7.2.1511 (Core) Release: 7.2.1511 Codename: Core
There is a dedicated website for ZFS on Linux where you can find the instructions on how to install it for various distributions. The instruction for CentOS/RHEL are quite easy. Download the repo files:
[root@centos7 ~] yum install http://download.zfsonlinux.org/epel/zfs-release$(rpm -E %dist).noarch.rpm Loaded plugins: fastestmirror zfs-release.el7.centos.noarch.rpm | 5.0 kB 00:00:00 Examining /var/tmp/yum-root-Uv79vc/zfs-release.el7.centos.noarch.rpm: zfs-release-1-3.el7.centos.noarch Marking /var/tmp/yum-root-Uv79vc/zfs-release.el7.centos.noarch.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package zfs-release.noarch 0:1-3.el7.centos will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================ Package Arch Version Repository Size ============================================================================================================================ Installing: zfs-release noarch 1-3.el7.centos /zfs-release.el7.centos.noarch 2.9 k Transaction Summary ============================================================================================================================ Install 1 Package Total size: 2.9 k Installed size: 2.9 k Is this ok [y/d/N]: y Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : zfs-release-1-3.el7.centos.noarch 1/1 Verifying : zfs-release-1-3.el7.centos.noarch 1/1 Installed: zfs-release.noarch 0:1-3.el7.centos Complete! [root@centos7 ~] gpg --quiet --with-fingerprint /etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux gpg: new configuration file `/root/.gnupg/gpg.conf' created gpg: WARNING: options in `/root/.gnupg/gpg.conf' are not yet active during this run pub 2048R/F14AB620 2013-03-21 ZFS on Linux Key fingerprint = C93A FFFD 9F3F 7B03 C310 CEB6 A9D5 A1C0 F14A B620 sub 2048R/99685629 2013-03-21
For the next step it depends if you want to go with DKMS or kABI-tracking kmod. I’ll go with kABI-tracking kmod and therefore will disable the DKMS repository and enable the kmod repository:
[root@centos7 ~] cat /etc/yum.repos.d/zfs.repo [zfs] name=ZFS on Linux for EL7 - dkms baseurl=http://download.zfsonlinux.org/epel/7/$basearch/ enabled=0 metadata_expire=7d gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux [zfs-kmod] name=ZFS on Linux for EL7 - kmod baseurl=http://download.zfsonlinux.org/epel/7/kmod/$basearch/ enabled=1 metadata_expire=7d gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux [zfs-source] name=ZFS on Linux for EL7 - Source baseurl=http://download.zfsonlinux.org/epel/7/SRPMS/ enabled=0 gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux [zfs-testing] name=ZFS on Linux for EL7 - dkms - Testing baseurl=http://download.zfsonlinux.org/epel-testing/7/$basearch/ enabled=0 metadata_expire=7d gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux [zfs-testing-kmod] name=ZFS on Linux for EL7 - kmod - Testing baseurl=http://download.zfsonlinux.org/epel-testing/7/kmod/$basearch/ enabled=0 metadata_expire=7d gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux [zfs-testing-source] name=ZFS on Linux for EL7 - Testing Source baseurl=http://download.zfsonlinux.org/epel-testing/7/SRPMS/ enabled=0 gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux [root@centos7 ~]
Installing ZFS from here on is just a matter of using yum:
[root@centos7 ~] yum install zfs Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.spreitzer.ch * extras: mirror.spreitzer.ch * updates: mirror.de.leaseweb.net zfs-kmod/x86_64/primary_db | 231 kB 00:00:01 Resolving Dependencies --> Running transaction check ---> Package zfs.x86_64 0:0.6.5.8-1.el7.centos will be installed --> Processing Dependency: zfs-kmod = 0.6.5.8 for package: zfs-0.6.5.8-1.el7.centos.x86_64 --> Processing Dependency: spl = 0.6.5.8 for package: zfs-0.6.5.8-1.el7.centos.x86_64 --> Processing Dependency: libzpool2 = 0.6.5.8 for package: zfs-0.6.5.8-1.el7.centos.x86_64 --> Processing Dependency: libzfs2 = 0.6.5.8 for package: zfs-0.6.5.8-1.el7.centos.x86_64 --> Processing Dependency: libuutil1 = 0.6.5.8 for package: zfs-0.6.5.8-1.el7.centos.x86_64 --> Processing Dependency: libnvpair1 = 0.6.5.8 for package: zfs-0.6.5.8-1.el7.centos.x86_64 --> Processing Dependency: libzpool.so.2()(64bit) for package: zfs-0.6.5.8-1.el7.centos.x86_64 --> Processing Dependency: libzfs_core.so.1()(64bit) for package: zfs-0.6.5.8-1.el7.centos.x86_64 --> Processing Dependency: libzfs.so.2()(64bit) for package: zfs-0.6.5.8-1.el7.centos.x86_64 --> Processing Dependency: libuutil.so.1()(64bit) for package: zfs-0.6.5.8-1.el7.centos.x86_64 --> Processing Dependency: libnvpair.so.1()(64bit) for package: zfs-0.6.5.8-1.el7.centos.x86_64 --> Running transaction check ---> Package kmod-zfs.x86_64 0:0.6.5.8-1.el7.centos will be installed --> Processing Dependency: spl-kmod for package: kmod-zfs-0.6.5.8-1.el7.centos.x86_64 ---> Package libnvpair1.x86_64 0:0.6.5.8-1.el7.centos will be installed ---> Package libuutil1.x86_64 0:0.6.5.8-1.el7.centos will be installed ---> Package libzfs2.x86_64 0:0.6.5.8-1.el7.centos will be installed ---> Package libzpool2.x86_64 0:0.6.5.8-1.el7.centos will be installed ---> Package spl.x86_64 0:0.6.5.8-1.el7.centos will be installed --> Running transaction check ---> Package kmod-spl.x86_64 0:0.6.5.8-1.el7.centos will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================ Package Arch Version Repository Size ============================================================================================================================ Installing: zfs x86_64 0.6.5.8-1.el7.centos zfs-kmod 334 k Installing for dependencies: kmod-spl x86_64 0.6.5.8-1.el7.centos zfs-kmod 110 k kmod-zfs x86_64 0.6.5.8-1.el7.centos zfs-kmod 665 k libnvpair1 x86_64 0.6.5.8-1.el7.centos zfs-kmod 35 k libuutil1 x86_64 0.6.5.8-1.el7.centos zfs-kmod 41 k libzfs2 x86_64 0.6.5.8-1.el7.centos zfs-kmod 123 k libzpool2 x86_64 0.6.5.8-1.el7.centos zfs-kmod 423 k spl x86_64 0.6.5.8-1.el7.centos zfs-kmod 29 k Transaction Summary ============================================================================================================================ Install 1 Package (+7 Dependent packages) Total download size: 1.7 M Installed size: 5.7 M Is this ok [y/d/N]: y Downloading packages: (1/8): kmod-spl-0.6.5.8-1.el7.centos.x86_64.rpm | 110 kB 00:00:01 (2/8): libnvpair1-0.6.5.8-1.el7.centos.x86_64.rpm | 35 kB 00:00:00 (3/8): libuutil1-0.6.5.8-1.el7.centos.x86_64.rpm | 41 kB 00:00:00 (4/8): kmod-zfs-0.6.5.8-1.el7.centos.x86_64.rpm | 665 kB 00:00:02 (5/8): libzfs2-0.6.5.8-1.el7.centos.x86_64.rpm | 123 kB 00:00:00 (6/8): libzpool2-0.6.5.8-1.el7.centos.x86_64.rpm | 423 kB 00:00:00 (7/8): spl-0.6.5.8-1.el7.centos.x86_64.rpm | 29 kB 00:00:00 (8/8): zfs-0.6.5.8-1.el7.centos.x86_64.rpm | 334 kB 00:00:00 ---------------------------------------------------------------------------------------------------------------------------- Total 513 kB/s | 1.7 MB 00:00:03 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : libuutil1-0.6.5.8-1.el7.centos.x86_64 1/8 Installing : libnvpair1-0.6.5.8-1.el7.centos.x86_64 2/8 Installing : libzpool2-0.6.5.8-1.el7.centos.x86_64 3/8 Installing : kmod-spl-0.6.5.8-1.el7.centos.x86_64 4/8 Installing : spl-0.6.5.8-1.el7.centos.x86_64 5/8 Installing : libzfs2-0.6.5.8-1.el7.centos.x86_64 6/8 Installing : kmod-zfs-0.6.5.8-1.el7.centos.x86_64 7/8 Installing : zfs-0.6.5.8-1.el7.centos.x86_64 8/8 Verifying : libnvpair1-0.6.5.8-1.el7.centos.x86_64 1/8 Verifying : libzfs2-0.6.5.8-1.el7.centos.x86_64 2/8 Verifying : zfs-0.6.5.8-1.el7.centos.x86_64 3/8 Verifying : spl-0.6.5.8-1.el7.centos.x86_64 4/8 Verifying : kmod-zfs-0.6.5.8-1.el7.centos.x86_64 5/8 Verifying : libzpool2-0.6.5.8-1.el7.centos.x86_64 6/8 Verifying : libuutil1-0.6.5.8-1.el7.centos.x86_64 7/8 Verifying : kmod-spl-0.6.5.8-1.el7.centos.x86_64 8/8 Installed: zfs.x86_64 0:0.6.5.8-1.el7.centos Dependency Installed: kmod-spl.x86_64 0:0.6.5.8-1.el7.centos kmod-zfs.x86_64 0:0.6.5.8-1.el7.centos libnvpair1.x86_64 0:0.6.5.8-1.el7.centos libuutil1.x86_64 0:0.6.5.8-1.el7.centos libzfs2.x86_64 0:0.6.5.8-1.el7.centos libzpool2.x86_64 0:0.6.5.8-1.el7.centos spl.x86_64 0:0.6.5.8-1.el7.centos Complete! [root@centos7 ~]
Be aware that the kernel modules are not loaded by default, so you have to do this on your own:
[root@centos7 ~] /sbin/modprobe zfs Last login: Wed Sep 28 11:04:21 2016 from 192.168.22.1 [postgres@centos7 ~]$ lsmod | grep zfs zfs 2713912 0 zunicode 331170 1 zfs zavl 15236 1 zfs zcommon 55411 1 zfs znvpair 93227 2 zfs,zcommon spl 92223 3 zfs,zcommon,znvpair [root@centos7 ~] zfs list no datasets available
For loading the modules automatically create a file under /etc/modules-load.d:
[root@centos7 ~] echo "zfs" > /etc/modules-load.d/zfs.conf [root@centos7 ~] cat /etc/modules-load.d/zfs.conf zfs
So far so good. Lets create a ZFS file system. I have two disks available for playing with ZFS (sdb and sdc):
[root@centos7 ~] ls -la /dev/sd* brw-rw----. 1 root disk 8, 0 Sep 28 11:14 /dev/sda brw-rw----. 1 root disk 8, 1 Sep 28 11:14 /dev/sda1 brw-rw----. 1 root disk 8, 2 Sep 28 11:14 /dev/sda2 brw-rw----. 1 root disk 8, 16 Sep 28 11:14 /dev/sdb brw-rw----. 1 root disk 8, 32 Sep 28 11:14 /dev/sdc
The first thing you have to do is to create a new zfs pool (I don’t care about the warnings, that is why I use the “-f” option below):
[root@centos7 ~] zpool create pgpool mirror /dev/sdb /dev/sdc invalid vdev specification use '-f' to override the following errors: /dev/sdb does not contain an EFI label but it may contain partition information in the MBR. /dev/sdc does not contain an EFI label but it may contain partition information in the MBR. [root@centos7 ~] zpool create pgpool mirror /dev/sdb /dev/sdc -f [root@centos7 ~] zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT pgpool 9.94G 65K 9.94G - 0% 0% 1.00x ONLINE - [root@centos7 ~] zpool status pgpool pool: pgpool state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM pgpool ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sdb ONLINE 0 0 0 sdc ONLINE 0 0 0 errors: No known data errors [root@centos7 ~] df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos-root 49G 1.7G 47G 4% / devtmpfs 235M 0 235M 0% /dev tmpfs 245M 0 245M 0% /dev/shm tmpfs 245M 4.3M 241M 2% /run tmpfs 245M 0 245M 0% /sys/fs/cgroup /dev/sda1 497M 291M 206M 59% /boot tmpfs 49M 0 49M 0% /run/user/1000 pgpool 9.7G 0 9.7G 0% /pgpool
What I did here is to create a mirrored pool over my two disks. The open zfs wiki has some performance tips for running PostgreSQL on ZFS as well as for other topics. Lets go with the recommendations:
[root@centos7 ~] zfs create pgpool/pgdata -o recordsize=8192 [root@centos7 ~] zfs set logbias=throughput pgpool/pgdata [root@centos7 ~] zfs set primarycache=all pgpool/pgdata [root@centos7 ~] zfs list NAME USED AVAIL REFER MOUNTPOINT pgpool 82K 9.63G 19.5K /pgpool pgpool/pgdata 19K 9.63G 19K /pgpool/pgdata
My new ZFS file system is ready and already mounted, cool. Lets change the permissions and list all the properties:
[root@centos7 ~] chown postgres:postgres /pgpool/pgdata [root@centos7 ~] zfs get all /pgpool/pgdata NAME PROPERTY VALUE SOURCE pgpool/pgdata type filesystem - pgpool/pgdata creation Wed Sep 28 11:31 2016 - pgpool/pgdata used 19K - pgpool/pgdata available 9.63G - pgpool/pgdata referenced 19K - pgpool/pgdata compressratio 1.00x - pgpool/pgdata mounted yes - pgpool/pgdata quota none default pgpool/pgdata reservation none default pgpool/pgdata recordsize 8K local pgpool/pgdata mountpoint /pgpool/pgdata default pgpool/pgdata sharenfs off default pgpool/pgdata checksum on default pgpool/pgdata compression off default pgpool/pgdata atime on default pgpool/pgdata devices on default pgpool/pgdata exec on default pgpool/pgdata setuid on default pgpool/pgdata readonly off default pgpool/pgdata zoned off default pgpool/pgdata snapdir hidden default pgpool/pgdata aclinherit restricted default pgpool/pgdata canmount on default pgpool/pgdata xattr on default pgpool/pgdata copies 1 default pgpool/pgdata version 5 - pgpool/pgdata utf8only off - pgpool/pgdata normalization none - pgpool/pgdata casesensitivity sensitive - pgpool/pgdata vscan off default pgpool/pgdata nbmand off default pgpool/pgdata sharesmb off default pgpool/pgdata refquota none default pgpool/pgdata refreservation none default pgpool/pgdata primarycache all default pgpool/pgdata secondarycache all default pgpool/pgdata usedbysnapshots 0 - pgpool/pgdata usedbydataset 19K - pgpool/pgdata usedbychildren 0 - pgpool/pgdata usedbyrefreservation 0 - pgpool/pgdata logbias throughput local pgpool/pgdata dedup off default pgpool/pgdata mlslabel none default pgpool/pgdata sync standard default pgpool/pgdata refcompressratio 1.00x - pgpool/pgdata written 19K - pgpool/pgdata logicalused 9.50K - pgpool/pgdata logicalreferenced 9.50K - pgpool/pgdata filesystem_limit none default pgpool/pgdata snapshot_limit none default pgpool/pgdata filesystem_count none default pgpool/pgdata snapshot_count none default pgpool/pgdata snapdev hidden default pgpool/pgdata acltype off default pgpool/pgdata context none default pgpool/pgdata fscontext none default pgpool/pgdata defcontext none default pgpool/pgdata rootcontext none default pgpool/pgdata relatime on temporary pgpool/pgdata redundant_metadata all default pgpool/pgdata overlay off default
Ready to deploy a PostgreSQL instance on it:
postgres@centos7:/home/postgres/ [pg954] initdb -D /pgpool/pgdata/ The files belonging to this database system will be owned by user "postgres". This user must also own the server process. The database cluster will be initialized with locales COLLATE: en_US.UTF-8 CTYPE: en_US.UTF-8 MESSAGES: en_US.UTF-8 MONETARY: de_CH.UTF-8 NUMERIC: de_CH.UTF-8 TIME: en_US.UTF-8 The default database encoding has accordingly been set to "UTF8". The default text search configuration will be set to "english". Data page checksums are disabled. fixing permissions on existing directory /pgpool/pgdata ... ok creating subdirectories ... ok selecting default max_connections ... 100 selecting default shared_buffers ... 128MB selecting dynamic shared memory implementation ... posix creating configuration files ... ok creating template1 database in /pgpool/pgdata/base/1 ... ok initializing pg_authid ... ok initializing dependencies ... ok creating system views ... ok loading system objects' descriptions ... ok creating collations ... ok creating conversions ... ok creating dictionaries ... ok setting privileges on built-in objects ... ok creating information schema ... ok loading PL/pgSQL server-side language ... ok vacuuming database template1 ... ok copying template1 to template0 ... ok copying template1 to postgres ... ok syncing data to disk ... ok WARNING: enabling "trust" authentication for local connections You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. Success. You can now start the database server using: pg_ctl -D /pgpool/pgdata/ -l logfile start
Startup:
postgres@centos7:/home/postgres/ [pg954] mkdir /pgpool/pgdata/pg_log postgres@centos7:/home/postgres/ [pg954] sed -i 's/logging_collector = off/logging_collector = on/g' /pgpool/pgdata/postgresql.conf postgres@centos7:/home/postgres/ [pg954] pg_ctl -D /pgpool/pgdata/ start postgres@centos7:/home/postgres/ [pg954] psql postgres psql (9.5.4 dbi services build) Type "help" for help. postgres=
Ready. Lets reboot and check if the ZFS file system is mounted automatically:
postgres@centos7:/home/postgres/ [pg954] df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos-root 49G 1.8G 47G 4% / devtmpfs 235M 0 235M 0% /dev tmpfs 245M 0 245M 0% /dev/shm tmpfs 245M 4.3M 241M 2% /run tmpfs 245M 0 245M 0% /sys/fs/cgroup /dev/sda1 497M 291M 206M 59% /boot tmpfs 49M 0 49M 0% /run/user/1000 postgres@centos7:/home/postgres/ [pg954] lsmod | grep zfs zfs 2713912 0 zunicode 331170 1 zfs zavl 15236 1 zfs zcommon 55411 1 zfs znvpair 93227 2 zfs,zcommon spl 92223 3 zfs,zcommon,znvpair
Gone. The kernel modules are loaded but the file system was not mounted. What to do?
[root@centos7 ~] zpool list no pools available [root@centos7 ~] zpool import pgpool [root@centos7 ~] zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT pgpool 9.94G 39.3M 9.90G - 0% 0% 1.00x ONLINE - [root@centos7 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos-root 49G 1.8G 47G 4% / devtmpfs 235M 0 235M 0% /dev tmpfs 245M 0 245M 0% /dev/shm tmpfs 245M 4.3M 241M 2% /run tmpfs 245M 0 245M 0% /sys/fs/cgroup /dev/sda1 497M 291M 206M 59% /boot tmpfs 49M 0 49M 0% /run/user/1000 pgpool 9.6G 0 9.6G 0% /pgpool pgpool/pgdata 9.7G 39M 9.6G 1% /pgpool/pgdata
Ok, how to auto mount?
[root@centos7 ~] systemctl enable zfs-mount [root@centos7 ~] systemctl enable zfs-import-cache [root@centos7 ~] reboot
I am not sure why this is necessary, should happen automatically.
PS: There is an interesting discussion about PostgreSQL on ZFS on the PostgreSQL performance mailing list currently.