dbi Blog

Quite some time ago I blogged about how you could build your customzized PostgreSQL container by using a Dockerfile and Docker build. In the meantime Red Hat replaced Docker in OpenShift and SUSE replaced Docker as well in CaaS. As a consequence there need to be other ways of building containers and one of them is buildah. You can use buildah to build from a Docker file as well, but in this post we will use a simple bash script to create the container.

We start be defining four variables that define PGDATA, the PostgreSQL major version, the full version string and the minor version which will be used to create our standard installation location (these will also go into the entrypoint, see below):

#!/bin/bash
_PGDATA="/u02/pgdata"
_PGMAJOR=12
_PGVERSION=12.0
_PGMINOR="db_0"

As mentioned in the beginning buildah will be used to create the container. For running the container we need something else, and that is podman. You can run the container buildah creates with plain Docker as well, if you want, as it is oci compliant but as Red Hat does not ship Docker anymore we will use the recommended way of doing it by using podman. So the natural next step in the script is do install buildah and podman:

dnf install -y buildah podman

Buildah can create containers from scratch, which means you start with a container that contains nothing except some meta data:

newcontainer=$(buildah from scratch)

Once we have the new scratch container it gets mounted so dnf can be used to install the packages we need into the container without actually using dnf in the container:

scratchmnt=$(buildah mount $newcontainer)
ls -la $scratchmnt
dnf install --installroot $scratchmnt --releasever 8 bash coreutils gcc openldap-devel platform-python-devel readline-devel bison flex perl-ExtUtils-Embed zlib-devel openssl-devel pam-devel libxml2-devel libxslt-devel bzip2 wget policycoreutils-python-utils make tar --setopt install_weak_deps=false --setopt=tsflags=nodocs --setopt=override_install_langs=en_US.utf8 -y

Using “buildah config” the container can be configured. Here it is about the author, environment variables, the default user and the entrypoint that will be used once the conatiner will be started:

buildah config --created-by "dbi services"  $newcontainer
buildah config --author "dbi services" --label name=dbiservices $newcontainer
buildah run $newcontainer groupadd postgres
buildah run $newcontainer useradd -g postgres -m postgres
buildah config --user postgres $newcontainer
buildah config --workingdir /home/postgres $newcontainer
buildah config --env PGDATABASE="" $newcontainer
buildah config --env PGUSERNAME="" $newcontainer
buildah config --env PGPASSWORD="" $newcontainer
buildah config --env PGDATA=${_PGDATA} $newcontainer
buildah config --env PGMAJOR=${_PGMAJOR} $newcontainer
buildah config --env PGMINOR=${_PGMINOR} $newcontainer
buildah config --env PGVERSION=${_PGVERSION} $newcontainer
buildah config --entrypoint /usr/bin/entrypoint.sh $newcontainer
buildah copy $newcontainer ./entrypoint.sh /usr/bin/entrypoint.sh
buildah run $newcontainer chmod +x /usr/bin/entrypoint.sh

What follows is basically installing PostgreSQL from source code:

buildah run --user root $newcontainer mkdir -p /u01 /u02
buildah run --user root $newcontainer chown postgres:postgres /u01 /u02
buildah run --user postgres $newcontainer wget https://ftp.postgresql.org/pub/source/v${_PGVERSION}/postgresql-${_PGVERSION}.tar.bz2 -O /home/postgres/postgresql-${_PGVERSION}.tar.bz2
buildah run --user postgres $newcontainer /usr/bin/bunzip2 /home/postgres/postgresql-${_PGVERSION}.tar.bz2
buildah run --user postgres $newcontainer /usr/bin/tar -xvf /home/postgres/postgresql-${_PGVERSION}.tar -C /home/postgres/
buildah run --user postgres $newcontainer /home/postgres/postgresql-12.0/configure --prefix=/u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR} --exec-prefix=/u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR} --bindir=/u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin --libdir=/u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/lib --includedir=/u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/include 
buildah run --user postgres $newcontainer /usr/bin/make -C /home/postgres all
buildah run --user postgres $newcontainer /usr/bin/make -C /home/postgres install
buildah run --user postgres $newcontainer /usr/bin/make -C /home/postgres/contrib install

Containers shoud be as small as possible so lets do some cleanup:

buildah run --user postgres $newcontainer /usr/bin/rm -rf /home/postgres/postgresql-${_PGVERSION}.tar
buildah run --user postgres $newcontainer /usr/bin/rm -rf /home/postgres/config
buildah run --user postgres $newcontainer /usr/bin/rm -rf /home/postgres/config.log
buildah run --user postgres $newcontainer /usr/bin/rm -rf /home/postgres/config.status
buildah run --user postgres $newcontainer /usr/bin/rm -rf /home/postgres/contrib
buildah run --user postgres $newcontainer /usr/bin/rm -rf /home/postgres/GNUmakefile
buildah run --user postgres $newcontainer /usr/bin/rm -rf /home/postgres/postgresql-12.0
buildah run --user postgres $newcontainer /usr/bin/rm -rf /home/postgres/src
buildah run --user postgres $newcontainer /usr/bin/rm -rf /home/postgres/doc
buildah run --user postgres $newcontainer /usr/bin/rm -rf /home/postgres/Makefile
buildah run --user postgres $newcontainer /usr/bin/rm -rf /home/postgres/.wget-hsts

When you want to run PostgreSQL inside a container you do not need any of the following binaries, so these can be cleaned as well:

buildah run --user postgres $newcontainer /usr/bin/rm -rf /u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin/vacuumlo
buildah run --user postgres $newcontainer /usr/bin/rm -rf /u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin/vacuumdb
buildah run --user postgres $newcontainer /usr/bin/rm -rf /u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin/reindexdb
buildah run --user postgres $newcontainer /usr/bin/rm -rf /u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin/pgbench
buildah run --user postgres $newcontainer /usr/bin/rm -rf /u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin/pg_waldump
buildah run --user postgres $newcontainer /usr/bin/rm -rf /u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin/pg_test_timing
buildah run --user postgres $newcontainer /usr/bin/rm -rf /u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin/pg_test_fsync
buildah run --user postgres $newcontainer /usr/bin/rm -rf /u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin/pg_standby
buildah run --user postgres $newcontainer /usr/bin/rm -rf /u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin/pg_restore
buildah run --user postgres $newcontainer /usr/bin/rm -rf /u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin/pg_recvlogical
buildah run --user postgres $newcontainer /usr/bin/rm -rf /u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin/pg_receivewal
buildah run --user postgres $newcontainer /usr/bin/rm -rf /u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin/pg_isready
buildah run --user postgres $newcontainer /usr/bin/rm -rf /u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin/pg_dumpall
buildah run --user postgres $newcontainer /usr/bin/rm -rf /u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin/pg_dump
buildah run --user postgres $newcontainer /usr/bin/rm -rf /u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin/pg_checksums
buildah run --user postgres $newcontainer /usr/bin/rm -rf /u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin/pg_basebackup
buildah run --user postgres $newcontainer /usr/bin/rm -rf /u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin/pg_archivecleanup
buildah run --user postgres $newcontainer /usr/bin/rm -rf /u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin/oid2name
buildah run --user postgres $newcontainer /usr/bin/rm -rf /u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin/dropuser
buildah run --user postgres $newcontainer /usr/bin/rm -rf /u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin/dropdb
buildah run --user postgres $newcontainer /usr/bin/rm -rf /u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin/createuser
buildah run --user postgres $newcontainer /usr/bin/rm -rf /u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin/createdb
buildah run --user postgres $newcontainer /usr/bin/rm -rf /u01/app/postgres/product/${_PGMAJOR}/${_PGMINOR}/bin/clusterdb

Last, but not least remove all the packages we do not require anymore and get rid of the dnf cache:

dnf remove --installroot $scratchmnt --releasever 8 gcc openldap-devel readline-devel bison flex perl-ExtUtils-Embed zlib-devel openssl-devel pam-devel libxml2-devel libxslt-devel bzip2 wget policycoreutils-python-utils make tar -y
dnf clean all -y --installroot $scratchmnt --releasever 8
# Clean up yum cache
if [ -d "${scratchmnt}" ]; then
rm -rf "${scratchmnt}"/var/cache/yum
fi
buildah unmount $newcontainer

Ready to publish the container:

buildah commit $newcontainer dbi-postgres

When you put all those steps into a script and run that you should see the just created container:

[root@doag2019 ~]$ buildah containers
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME
47946e4b4fc8     *                  scratch                          working-container
[root@doag2019 ~]$

… but now we also have a new image that can be started:

IMAGE NAME                                               IMAGE TAG            IMAGE ID             CREATED AT             SIZE
localhost/dbi-postgres                                   latest               dfcd3e8d5273         Oct 13, 2019 13:22     461 MB

Once we start that the entrypoint will be executed:

#!/bin/bash
# this are the environment variables which need to be set
PGDATA=${PGDATA}/${PGMAJOR}
PGHOME="/u01/app/postgres/product/${PGMAJOR}/${PGMINOR}"
PGAUTOCONF=${PGDATA}/postgresql.auto.conf
PGHBACONF=${PGDATA}/pg_hba.conf
PGDATABASENAME=${PGDATABASE}
PGUSERNAME=${PGUSERNAME}
PGPASSWD=${PGPASSWORD}
# create the database and the user
_pg_create_database_and_user()
{
${PGHOME}/bin/psql -c "create user ${PGUSERNAME} with login password '${PGPASSWD}'" postgres
${PGHOME}/bin/psql -c "create database ${PGDATABASENAME} with owner = ${PGUSERNAME}" postgres
${PGHOME}/bin/psql -c "create extension pg_stat_statements" postgres
}
# start the PostgreSQL instance
_pg_prestart()
{
${PGHOME}/bin/pg_ctl -D ${PGDATA} -w start
}
# Start PostgreSQL without detaching 
_pg_start()
{
exec ${PGHOME}/bin/postgres "-D" "${PGDATA}"
}
# stop the PostgreSQL instance
_pg_stop()
{
${PGHOME}/bin/pg_ctl -D ${PGDATA} stop -m fast
}
# initdb a new cluster
_pg_initdb()
{
${PGHOME}/bin/initdb -D ${PGDATA} --data-checksums
}
# adjust the postgresql parameters
_pg_adjust_config() {
if [ -z $PGMEMORY ]; then MEM="128MB"
else                      MEM=$PGMEMORY; fi
# PostgreSQL parameters
echo "shared_buffers='$MEM'" >> ${PGAUTOCONF}
echo "effective_cache_size='128MB'" >> ${PGAUTOCONF}
echo "listen_addresses = '*'" >> ${PGAUTOCONF}
echo "logging_collector = 'off'" >> ${PGAUTOCONF}
echo "log_truncate_on_rotation = 'on'" >> ${PGAUTOCONF}
echo "log_line_prefix = '%m - %l - %p - %h - %u@%d '" >> ${PGAUTOCONF}
echo "log_directory = 'pg_log'" >> ${PGAUTOCONF}
echo "log_min_messages = 'WARNING'" >> ${PGAUTOCONF}
echo "log_autovacuum_min_duration = '60s'" >> ${PGAUTOCONF}
echo "log_min_error_statement = 'NOTICE'" >> ${PGAUTOCONF}
echo "log_min_duration_statement = '30s'" >> ${PGAUTOCONF}
echo "log_checkpoints = 'on'" >> ${PGAUTOCONF}
echo "log_statement = 'none'" >> ${PGAUTOCONF}
echo "log_lock_waits = 'on'" >> ${PGAUTOCONF}
echo "log_temp_files = '0'" >> ${PGAUTOCONF}
echo "log_timezone = 'Europe/Zurich'" >> ${PGAUTOCONF}
echo "log_connections=on" >> ${PGAUTOCONF}
echo "log_disconnections=on" >> ${PGAUTOCONF}
echo "log_duration=off" >> ${PGAUTOCONF}
echo "client_min_messages = 'WARNING'" >> ${PGAUTOCONF}
echo "wal_level = 'replica'" >> ${PGAUTOCONF}
echo "wal_compression=on" >> ${PGAUTOCONF}
echo "max_replication_slots=20" >> ${PGAUTOCONF}
echo "max_wal_senders=20" >> ${PGAUTOCONF}
echo "hot_standby_feedback = 'on'" >> ${PGAUTOCONF}
echo "cluster_name = '${PGDATABASENAME}'" >> ${PGAUTOCONF}
echo "max_replication_slots = '10'" >> ${PGAUTOCONF}
echo "work_mem=8MB" >> ${PGAUTOCONF}
echo "maintenance_work_mem=64MB" >> ${PGAUTOCONF}
echo "shared_preload_libraries='pg_stat_statements'" >> ${PGAUTOCONF}
echo "autovacuum_max_workers=6" >> ${PGAUTOCONF}
echo "autovacuum_vacuum_scale_factor=0.1" >> ${PGAUTOCONF}
echo "autovacuum_vacuum_threshold=50" >> ${PGAUTOCONF}
echo "archive_mode=on" >> ${PGAUTOCONF}
echo "archive_command='/bin/true'" >> ${PGAUTOCONF}
# Authentication settings in pg_hba.conf
echo "host    all             all             0.0.0.0/0            md5"  >> ${PGHBACONF}
}
# initialize and start a new cluster
_pg_init_and_start()
{
# initialize a new cluster
_pg_initdb
# set params and access permissions
_pg_adjust_config
# start the new cluster
_pg_prestart
# set username and password
_pg_create_database_and_user
# restart database with correct pid
_pg_stop
_pg_start
}
# check if $PGDATA exists
if [ -e ${PGDATA} ]; then
# when $PGDATA exists we need to check if there are files
# because when there are files we do not want to initdb
if [ -e "${DEBUG}" ]; then
/bin/bash
elif [ -e "${PGDATA}/base" ]; then
# when there is the base directory this
# probably is a valid PostgreSQL cluster
# so we just start it
_pg_start
else
# when there is no base directory then we
# should be able to initialize a new cluster
# and then start it
_pg_init_and_start
fi
else
# create PGDATA
mkdir -p ${PGDATA}
# initialze and start the new cluster
_pg_init_and_start
fi

Starting that up using podman:

[root@doag2019 ~]$ podman run -e PGDATABASE=test -e PGUSERNAME=test -e PGPASSWORD=test --detach -p 5432:5432 localhost/dbi-postgres
f933df8216de83b3c2243860ace02f231748a05273c16d3ddb0308231004552f
CONTAINER ID  IMAGE                          COMMAND               CREATED             STATUS             PORTS                   NAMES
f933df8216de  localhost/dbi-postgres:latest  /bin/sh -c /usr/b...  About a minute ago  Up 59 seconds ago  0.0.0.0:5432->5432/tcp  nervous_leavitt

… and connecting from the host system:

[root@doag2019 ~]$ psql -p 5432 -h localhost -U test test
Password for user test:
psql (10.6, server 12.0)
WARNING: psql major version 10, server major version 12.
Some psql features might not work.
Type "help" for help.
test=> select version();
version
--------------------------------------------------------------------------------------------------------
PostgreSQL 12.0 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 8.2.1 20180905 (Red Hat 8.2.1-3), 64-bit
(1 row)
test=> q

One you have that scripted and ready it is a very convinient way for creating images. What I like most is, that you can make changes afterwards without starting from scratch:

[root@doag2019 ~]$ podman inspect localhost/dbi-postgres
[
{
"Id": "dfcd3e8d5273116e5678806dfe7bbf3ca2276549db73e62f27b967673df8084c",
"Digest": "sha256:b2d65e569becafbe64e8bcb6d49b065188411f596c04dea2cf335f677e2db68e",
"RepoTags": [
"localhost/dbi-postgres:latest"
],
"RepoDigests": [
"localhost/dbi-postgres@sha256:b2d65e569becafbe64e8bcb6d49b065188411f596c04dea2cf335f677e2db68e"
],
"Parent": "",
"Comment": "",
"Created": "2019-10-13T11:22:15.096957689Z",
"Config": {
"User": "postgres",
"Env": [
"PGDATABASE=",
"PGUSERNAME=",
"PGPASSWORD=",
"PGDATA=/u02/pgdata",
"PGMAJOR=12",
"PGMINOR=db_0",
"PGVERSION=12.0"
],
"Entrypoint": [
"/bin/sh",
"-c",
"/usr/bin/entrypoint.sh"
],
"WorkingDir": "/home/postgres",
"Labels": {
"name": "dbiservices"
}
},
"Version": "",
"Author": "dbiservices",
"Architecture": "amd64",
"Os": "linux",
"Size": 460805033,
"VirtualSize": 460805033,
"GraphDriver": {
"Name": "overlay",
"Data": {
"MergedDir": "/var/lib/containers/storage/overlay/89de699f19781bb61eec12cf61a097a9daa31d7725fc3c078c76d0d6291cb074/merged",
"UpperDir": "/var/lib/containers/storage/overlay/89de699f19781bb61eec12cf61a097a9daa31d7725fc3c078c76d0d6291cb074/diff",
"WorkDir": "/var/lib/containers/storage/overlay/89de699f19781bb61eec12cf61a097a9daa31d7725fc3c078c76d0d6291cb074/work"
}
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:89de699f19781bb61eec12cf61a097a9daa31d7725fc3c078c76d0d6291cb074"
]
},
"Labels": {
"name": "dbiservices"
},
"Annotations": {},
"ManifestType": "application/vnd.oci.image.manifest.v1+json",
"User": "postgres",
"History": [
{
"created": "2019-10-13T11:22:15.096957689Z",
"created_by": "dbi services",
"author": "dbiservices"
}
]
}
]

Assume we want to add a new environment variable. All we need to do is this:

[root@doag2019 ~]$ buildah containers
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME
47946e4b4fc8     *                  scratch                          working-container
[root@doag2019 ~]$ buildah config --env XXXXXXX="xxxxxxxx" 47946e4b4fc8
[root@doag2019 ~]$ buildah commit 47946e4b4fc8 dbi-postgres
Getting image source signatures
Skipping fetch of repeat blob sha256:9b74f2770486cdb56539b4a112b95ad7e10aced3a2213d33878f8fd736b5c684
Copying config sha256:e2db86571bfa2e64e6079077fe023e38a07544ccda529ba1c3bfc04984f2ac74
606 B / 606 B [============================================================] 0s
Writing manifest to image destination
Storing signatures
e2db86571bfa2e64e6079077fe023e38a07544ccda529ba1c3bfc04984f2ac74

The new image with the new variable is ready:

[root@doag2019 ~]$ buildah images
IMAGE NAME                                               IMAGE TAG            IMAGE ID             CREATED AT             SIZE
                                                                              dfcd3e8d5273         Oct 13, 2019 13:22     461 MB
localhost/dbi-postgres                                   latest               e2db86571bfa         Oct 13, 2019 13:52     461 MB
[root@doag2019 ~]$ buildah inspect localhost/dbi-postgres
...
"Env": [
"PGDATABASE=",
"PGUSERNAME=",
"PGPASSWORD=",
"PGDATA=/u02/pgdata",
"PGMAJOR=12",
"PGMINOR=db_0",
"PGVERSION=12.0",
"XXXXXXX=xxxxxxxx"
],
...

Nice. If you are happy with the image the scratch container can be deleted.

Post Views: 862

Creating a customized PostgreSQL container using buildah

Daniel Westermann

Leave a Reply:

Related blog articles

Daniel Westermann

Leave a Reply:

Related blog articles

A first look at openSUSE Leap Micro 6.0

Another file system for Linux: bcachefs (3) – Mirroring/Replicas

Another file system for Linux: bcachefs (2) – multi device file systems

Another file system for Linux: bcachefs (1) – basics