Infrastructure at your Service

During our last SwissPUG online meeting there was a presentation about monitoring PostgreSQL with temBoard. A question that came up afterwards was: How to install that on SLES 15 as there are only packages for Debian and CentOS/RHEL. As temBoard is written in Python you can also install it using pip. That requires a bit more work compared to the package method, but it is perfectly fine. The only downside (but this is true for the packaged version as well) is, that you need Python 2.7 for the WebUI, and this is already end of life. So, lets give it a try on a fresh SLES 15 minimal installation.

As mentioned just before we’ll start with a minimal installation, as this is intended to be a step by step guide:

sles15webui:~ $ cat /etc/os-release 
NAME="SLES"
VERSION="15-SP2"
VERSION_ID="15.2"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP2"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp2"

The minimal installation of SLES 15 comes with Python 3.6.13 and this is not what we need:

sles15webui:~ $ python3 --version
Python 3.6.13

For getting Python 2.7 onto the system, the corresponding repository needs to be enabled:

sles15webui:~ $ SUSEConnect --product sle-module-python2/15.2/x86_64
Registering system to SUSE Customer Center

Updating system details on https://scc.suse.com ...

Activating sle-module-python2 15.2 x86_64 ...
-> Adding service to system ...
-> Installing release package ...

Successfully registered system

Having that ready, we can install the requirements for temBoard:

sles15webui:~ $ zypper in python2 python2-pip
Refreshing service 'Basesystem_Module_15_SP2_x86_64'.
Refreshing service 'Python_2_Module_15_SP2_x86_64'.
Refreshing service 'SUSE_Linux_Enterprise_Server_15_SP2_x86_64'.
Refreshing service 'Server_Applications_Module_15_SP2_x86_64'.
...
sles15webui:~ $ python2 --version
Python 2.7.18

temBoard requires a PostgreSQL instance up and running as a repository database, and SLES 15 comes with PostgreSQL 13 already packaged:

sles15webui:~ $ zypper search postgres | grep 13
  | postgresql13                 | Basic Clients and Utilities for PostgreSQL                              | srcpackage
  | postgresql13                 | Basic Clients and Utilities for PostgreSQL                              | package
  | postgresql13-contrib         | Contributed Extensions and Additions to PostgreSQL                      | package
  | postgresql13-devel           | PostgreSQL client development header files and libraries                | package
  | postgresql13-docs            | HTML Documentation for PostgreSQL                                       | package
  | postgresql13-plperl          | The PL/Tcl, PL/Perl, and  PL/Python procedural languages for PostgreSQL | package
  | postgresql13-plpython        | The PL/Python Procedural Languages for PostgreSQL                       | package
  | postgresql13-pltcl           | PL/Tcl Procedural Language for PostgreSQL                               | package
  | postgresql13-server          | The Programs Needed to Create and Run a PostgreSQL Server               | package
  | postgresql13-server-devel    | PostgreSQL server development header files and utilities                | package

Either use that to install PostgreSQL and create a cluster, or install PostgreSQL from source code. I’ll skip these steps here and my PostgreSQL instance is already running:

[email protected]:/home/postgres/ [tb] psql
psql (13.2)
Type "help" for help.

postgres=# select version();
                                      version                                       
------------------------------------------------------------------------------------
 PostgreSQL 13.2 on x86_64-pc-linux-gnu, compiled by gcc (SUSE Linux) 7.5.0, 64-bit
(1 row)

The installation of temBoard is quite simple, all you have to do is this:

[email protected]:/home/postgres/ [tb] sudo pip2 install temboard psycopg2-binary
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting temboard
...
    Running setup.py install for tornado ... done
  WARNING: The script mako-render is installed in '/usr/local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script alembic is installed in '/usr/local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts temboard and temboard-migratedb are installed in '/usr/local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed Mako MarkupSafe alembic backports-abc configparser contextlib2 futures importlib-metadata pathlib2 psycopg2-binary python-dateutil python-editor scandir singledispatch sqlalchemy temboard tornado zipp

As you can see from the last lines above, temBoard gets installed into “/usr/local/bin”:

[email protected]:/home/postgres/ [tb] ls /usr/local/bin/
alembic  mako-render  temboard  temboard-migratedb

The other important directory is “/usr/local/share/temboard/”, this one contains all the scripts:

[email protected]:/home/postgres/ [tb] ls /usr/local/share/temboard/
auto_configure.sh  create_repository.sh  purge.sh  quickstart  sql

The “auto_configure.sh” script is the one which does all the work. You can either go with the default or adjust the parameters as you like, e.g.:

pos[email protected]:/home/postgres/ [tb] sudo su -
sles15webui:~ $ export ETCDIR=/u01/app/postgres/local/dmk/etc
sles15webui:~ $ export VARDIR=/u01/app/postgres/local/dmk/bin
sles15webui:~ $ export LOGDIR=/u01/app/postgres/local/dmk/log
sles15webui:~ $ export LOGFILE=/u01/app/postgres/local/dmk/log/temboard-auto-configure.log

The auto configuration script needs to connect to your PostgreSQL instance, so make sure that this works before starting:

sles15webui:~ $ export PATH=/u01/app/postgres/product/13/db_2/bin:$PATH
sles15webui:~ $ export PGHOST=/tmp
sles15webui:~ $ sudo -Eu postgres psql
could not change directory to "/root": Permission denied
psql (13.2)
Type "help" for help.

If that works just execute the script:

sles15webui:~ $ /usr/local/share/temboard/auto_configure.sh
Creating system user temBoard.
Configuring temboard in /u01/app/postgres/local/dmk/etc.
Generating self-signed certificate.
Creating Postgres user, database and schema.
Failure. See /u01/app/postgres/local/dmk/log/temboard-auto-configure.log for details.

… and that fails. Checking the log file, the reason is that “temboard-migratedb” is not found:

CREATE DATABASE
./create_repository.sh: line 17: temboard-migratedb: command not found
./create_repository.sh: line 18: temboard-migratedb: command not found

The easy fix is to update the create_repository.sh script:

if ! /usr/local/bin/temboard-migratedb check ; then
    /usr/local/bin/temboard-migratedb upgrade
    psql="psql -aw --set ON_ERROR_STOP=on --pset pager=off"
    if [ -n "${DEV-}" ] ; then
        $psql -f $SQLDIR/dev-fixture.sql
    fi
fi

Running the auto configuration once more, and it succeeds:

sles15webui:~ $ /usr/local/share/temboard/auto_configure.sh
Configuring temboard in /u01/app/postgres/local/dmk/etc.
Creating Postgres user, database and schema.

Success. You can now start temboard using:

    systemctl start temboard

Remember to replace default admin user!!!

A systemd service was created automatically:

sles15webui:~ $ systemctl status temboard
● temboard.service - temBoard Web UI
   Loaded: loaded (/usr/local/lib/systemd/system/temboard.service; enabled; vendor preset: disabled)
   Active: inactive (dead)
sles15webui:~ $ cat /usr/local/lib/systemd/system/temboard.service
[Unit]
Description=temBoard Web UI
After=network.target

[Service]
Type=simple
User=temboard
Group=temboard
ExecStart=/usr/bin/env SYSTEMD=1 temboard -c /etc/temboard/temboard.conf

[Install]
WantedBy=multi-user.target

The issue with that service is, that the configuration file does not exist:

sles15webui:~ $ cat /etc/temboard/temboard.conf
cat: /etc/temboard/temboard.conf: No such file or directory

Because we’ve set the environment variables before doing the temBoard installation, our configuration file is here:

sles15webui:~ $ cat /u01/app/postgres/local/dmk/etc/temboard.conf 
# Configuration initiated by /usr/local/share/temboard/auto_configure.sh on Thu Apr  1 09:17:58 CEST 2021
#
# See https://temboard.rtfd.io/ for details on configuration
# possibilities.

[temboard]
ssl_cert_file = /etc/ssl/certs/temboard.pem
ssl_key_file = /etc/ssl/private/temboard.key
cookie_secret = d11bcef48a3f6bc6de09cb3e39b88b99b2fae98b4d3dbfe501019efdcac681044be29eed3907b0cbe12dc64d49923f50f262b33fe1d154cf9b8609f2770edaef
home = /u01/app/postgres/local/dmk/bin

[repository]
host = /tmp
port = 5432
user = temboard
password = 4ebe5cd9715718780702420436cbd268
dbname = temboard

[logging]
method = stderr
level = INFO

[monitoring]
# purge_after = 365

[statements]
# purge_after = 7

Let’s try to start temBoard using this configuration manually as root:

sles15webui:~ $ /usr/bin/env SYSTEMD=1 temboard -c /u01/app/postgres/local/dmk/etc/temboard.conf
 INFO: Starting temboard 7.6.
 INFO: Running on SLES 15-SP2.
 INFO: Using Python 2.7.18 (/usr/bin/python2).
 INFO: Using Psycopg2 2.8.6 (dt dec pq3 ext lo64), Tornado 5.1.1 and SQLAlchemy 1.4.4
Loaded plugin 'activity'.
Loaded plugin 'dashboard'.
Loaded plugin 'monitoring'.
Loaded plugin 'pgconf'.
...

This works and accessing the UI over https://[IP]:8888 works as well:

So, we need to fix the systemd unit file to use the correct temBoard configuration file like this:

sles15webui:~ $ cat /usr/local/lib/systemd/system/temboard.service
[Unit]
Description=temBoard Web UI
After=network.target

[Service]
Type=simple
User=temboard
Group=temboard
ExecStart=/usr/bin/env SYSTEMD=1 temboard -c /u01/app/postgres/local/dmk/etc/temboard.conf

[Install]
WantedBy=multi-user.target
sles15webui:~ $ systemctl daemon-reload

Lets try to start with systemd:

sles15webui:~ $ systemctl start temboard
sles15webui:~ $ systemctl status temboard
● temboard.service - temBoard Web UI
   Loaded: loaded (/usr/local/lib/systemd/system/temboard.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2021-04-01 09:49:17 CEST; 3s ago
  Process: 28629 ExecStart=/usr/bin/env SYSTEMD=1 temboard -c /u01/app/postgres/local/dmk/etc/temboard.conf (code=exited, status=1/FAILURE)
 Main PID: 28629 (code=exited, status=1/FAILURE)

Apr 01 09:49:16 sles15webui systemd[1]: Started temBoard Web UI.
Apr 01 09:49:17 sles15webui env[28629]:  INFO: Starting temboard 7.6.
Apr 01 09:49:17 sles15webui env[28629]: ERROR: Invalid temboard_ssl_key_file from config: /etc/ssl/private/temboard.key: File not found...
Apr 01 09:49:17 sles15webui env[28629]: CRITI: Failed to load configuration.
Apr 01 09:49:17 sles15webui systemd[1]: temboard.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 09:49:17 sles15webui systemd[1]: temboard.service: Unit entered failed state.
Apr 01 09:49:17 sles15webui systemd[1]: temboard.service: Failed with result 'exit-code'.

Next issue, the key file can not be found (the service uses the tembaord user to start tembaord). The file actually is there, but the temboard user does not have access to it:

sles15webui:~ $ ls -la /etc/ssl/private/temboard.key
-rw------- 1 root root 1704 Apr  1 08:29 /etc/ssl/private/temboard.key

As I did not want to open permissions to the SSL configuration in /etc, I’ve copied that over:

sles15webui:~ $ mv /etc/ssl/private/temboard.key /u01/app/postgres/local/dmk/etc/
sles15webui:~ $ chown temboard:temboard /u01/app/postgres/local/dmk/etc/temboard.key 

Adjusted the temboard configuration with the new location of the key file:

sles15webui:~ $ cat /u01/app/postgres/local/dmk/etc/temboard.conf | grep key
ssl_key_file = /u01/app/postgres/local/dmk/etc/temboard.key

Started again:

sles15webui:~ $ systemctl start temboard
sles15webui:~ $ systemctl status temboard
● temboard.service - temBoard Web UI
   Loaded: loaded (/usr/local/lib/systemd/system/temboard.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2021-04-01 09:54:50 CEST; 2s ago
  Process: 28792 ExecStart=/usr/bin/env SYSTEMD=1 temboard -c /u01/app/postgres/local/dmk/etc/temboard.conf (code=exited, status=1/FAILURE)
 Main PID: 28792 (code=exited, status=1/FAILURE)

Apr 01 09:54:50 sles15webui env[28792]:     now=datetime.utcnow()
Apr 01 09:54:50 sles15webui env[28792]:   File "/usr/local/lib/python2.7/site-packages/temboardui/toolkit/tasklist/sqlite3_engine.py", line 253, in recover
Apr 01 09:54:50 sles15webui env[28792]:     raise StorageEngineError("Could not recover tasks.")
Apr 01 09:54:50 sles15webui env[28792]: StorageEngineError: Could not recover tasks.
Apr 01 09:54:50 sles15webui env[28792]: temboardui version is 7.6.
Apr 01 09:54:50 sles15webui env[28792]: This is a bug!
Apr 01 09:54:50 sles15webui env[28792]: Please report traceback to https://github.com/dalibo/temboard/issues! Thanks!
Apr 01 09:54:50 sles15webui systemd[1]: temboard.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 09:54:50 sles15webui systemd[1]: temboard.service: Unit entered failed state.
Apr 01 09:54:50 sles15webui systemd[1]: temboard.service: Failed with result 'exit-code'.

… and it again fails. Lets try to start manually with the temBoard user:

sles15webui:~ $ su - temboard
This account is currently not available.
sles15webui:~ $ usermod -s /bin/bash temboard
sles15webui:~ $ su - temboard
attempt to write a readonly database
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/temboardui/toolkit/tasklist/sqlite3_engine.py", line 244, in recover
    (st_aborted, datetime_to_epoch(now), st_doing)
OperationalError: attempt to write a readonly database
Unhandled error:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/temboardui/toolkit/app.py", line 298, in entrypoint
    retcode = self.main(argv, environ)
  File "/usr/local/lib/python2.7/site-packages/temboardui/__main__.py", line 395, in main
    self.scheduler.apply_config()
  File "/usr/local/lib/python2.7/site-packages/temboardui/__main__.py", line 143, in apply_config
    super(SchedulerService, self).apply_config()
  File "/usr/local/lib/python2.7/site-packages/temboardui/toolkit/taskmanager.py", line 724, in apply_config
    self.scheduler.setup_task_list()
  File "/usr/local/lib/python2.7/site-packages/temboardui/toolkit/taskmanager.py", line 482, in setup_task_list
    self.task_list.recover()
  File "/usr/local/lib/python2.7/site-packages/temboardui/toolkit/taskmanager.py", line 196, in recover
    now=datetime.utcnow()
  File "/usr/local/lib/python2.7/site-packages/temboardui/toolkit/tasklist/sqlite3_engine.py", line 253, in recover
    raise StorageEngineError("Could not recover tasks.")
StorageEngineError: Could not recover tasks.
temboardui version is 7.6.
This is a bug!
Please report traceback to https://github.com/dalibo/temboard/issues! Thanks!

The issue is this: “attempt to write a readonly database”. As there is no information where that database (seems to be SQLite) is located, lets use strace to spot the location:

[email protected]:~> strace -f /usr/bin/env SYSTEMD=1 temboard -c /u01/app/postgres/local/dmk/etc/temboard.conf 

Digging through the strace output, the database is this one: /u01/app/postgres/local/dmk/bin/server_tasks.db. Because we started as root before, this file has the woring permissions (this is not the fault of temBoard):

sles15webui:~$  ls -la /u01/app/postgres/local/dmk/bin/server_tasks.db
-rw-r--r-- 1 root root 12288 Apr  1 09:45 /u01/app/postgres/local/dmk/bin/server_tasks.db

Lets fix this, and try again:

sles15webui:~ $ chown temboard:temboard /u01/app/postgres/local/dmk/bin/server_tasks.db
sles15webui:~ $ systemctl status temboard
● temboard.service - temBoard Web UI
   Loaded: loaded (/usr/local/lib/systemd/system/temboard.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-04-01 10:15:22 CEST; 3s ago
 Main PID: 1064 (temboard)
    Tasks: 5
   CGroup: /system.slice/temboard.service
           ├─1064 temboard: web
           ├─1071 temboard: worker pool
           └─1072 temboard: scheduler

Apr 01 10:15:22 sles15webui env[1064]: Loaded plugin 'pgconf'.
Apr 01 10:15:22 sles15webui env[1064]: Loaded plugin 'maintenance'.
Apr 01 10:15:22 sles15webui env[1064]: Loaded plugin 'statements'.
Apr 01 10:15:22 sles15webui env[1064]: temBoard database is up-to-date.
Apr 01 10:15:22 sles15webui env[1064]: Starting web.
Apr 01 10:15:22 sles15webui env[1064]: Serving temboardui on https://0.0.0.0:8888
Apr 01 10:15:22 sles15webui env[1064]: Starting scheduler.
Apr 01 10:15:22 sles15webui env[1064]: Starting worker pool.
Apr 01 10:15:23 sles15webui env[1064]: Starting collector scheduler worker.
Apr 01 10:15:23 sles15webui env[1064]: End of collector scheduler worker.

Now we are fine, temBoard is started with the correct user with systemd, login (admin/admin) works as well:

This was kind of a brute force method for getting it installed, but if you plan that well, it should not be an issue if you follow these steps:

  • Prepare your own self signed certificate and make sure the temBoard user can access it
  • Prepare your own systemd configuration file
  • Prepare your own temBoard configuration file

In the next post we’ll install the temBoard agent on another SLES 15 machine, and actually want to monitor a PostgreSQL instance.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Daniel Westermann
Daniel Westermann

Principal Consultant & Technology Leader Open Infrastructure