Infrastructure at your Service

Daniel Westermann

SQL Monitoring in PostgreSQL (3) – pg_activity

The last posts looked at how the logging system and the pg_stat_statements extension can be used to monitor sql statements in PostgreSQL. This post will introduce pg_activity which is very similar to htop.

There are some dependencies which need to be installed before we can start installing pg_activity. The first one is python. As I am on a redhat based distribution this is quite easy:

[root@oel7 ~] yum install -y python

Then we need to install psycopg which is the database adapter for PostgreSQL for the python language (Note: if you have installed PostgreSQL not in the default location edit the setup.cfg script and provide the path to pg_config. Otherwise the install will fail):

postgres@oel7:/var/tmp/ [dummy] tar -axf psycopg2-2.6.1.tar.gz
postgres@oel7:/var/tmp/ [dummy] cd psycopg2-2.6.1
postgres@oel7:/var/tmp/psycopg2-2.6.1/ [dummy] python setup.py build
postgres@oel7:/var/tmp/psycopg2-2.6.1/ [dummy] sudo python setup.py install

The next (and last) thing we need to have available is psutils, a python library for querying os statistics:

postgres@oel7:/var/tmp/ [dummy] tar -axf psutil-3.3.0.tar.gz
postgres@oel7:/var/tmp/ [dummy] cd psutil-3.3.0
postgres@oel7:/var/tmp/ [dummy] sudo python setup.py install

That’s it. Now we can install pg_activity:

postgres@oel7:/var/tmp/ [dummy] unzip pg_activity-master.zip
postgres@oel7:/var/tmp/ [dummy] cd pg_activity-master
postgres@oel7:/var/tmp/ [dummy] sudo python setup.py install

Quite easy. Lets see what we can do with it. If you are locally on the server where your PostgreSQL instance runs you can just start pg_activity (I fired a sql statement so that you can see at least one sql in the screenshot):

postgres@oel7:/home/postgres/ [PG3] pg_activity

pg_activity_1

There is a nice summary on the top like in top/htop. The different statements which are currently executing are displayed below.

Hitting “h” for help shows you the various options:
pg_activity_2

The “F1/2/3″ switches are very nice when you want to display blocking queries only, running queries only or waiting queries only. Another great features is that you do not need to install pg_activity on the server where PostgreSQL is running. The same connections options as in e.g. psql are there so that you can connect to any remote PostgreSQL instance you have access to:

postgres@oel7:/home/postgres/ [PG3] pg_activity --help
Usage: pg_activity [options]

htop like application for PostgreSQL server activity monitoring.

Options:
  --version             show program's version number and exit
  -U USERNAME, --username=USERNAME
                        Database user name (default: "postgres").
  -p PORT, --port=PORT  Database server port (default: "5432").
  -h HOSTNAME, --host=HOSTNAME
                        Database server host or socket directory
                        (default: "localhost").
  -d DBNAME, --dbname=DBNAME
                        Database name to connect to (default: "postgres").
  -C, --no-color        Disable color usage.
  --blocksize=BLOCKSIZE
                        Filesystem blocksize (default: 4096)
  --rds                 Enable support for AWS RDS
  --help                Show this help message and exit.
  --debug               Enable debug mode for traceback tracking.

  Display Options, you can exclude some columns by using them :
    --no-database       Disable DATABASE.
    --no-user           Disable USER.
    --no-client         Disable CLIENT.
    --no-cpu            Disable CPU%.
    --no-mem            Disable MEM%.
    --no-read           Disable READ/s.
    --no-write          Disable WRITE/s.
    --no-time           Disable TIME+.
    --no-wait           Disable W.

Conclusion: pg_activity is small but very useful tool for monitoring an PostgreSQL instance. In the next posts I’ll look into some more feature rich monitoring solutions that are around for monitoring PostgreSQL instances.

 

7 Comments

  • Pavan says:

    Well about this extension. Actually it’s a very great explanation but the problem I’m facing is the queries are getting refreshed in the shot span unable to read it completely. So what should I do to get all the queries executed as a list or atleast increase the refresh interval

     
    • Daniel Westermann says:

      Hi Pavan,

      when go to help “h” there are options:

      +: increase refresh time (max:3)
      “-: decrease refresh time (min:1)”

      You can increase the refresh interval to 3 seconds, not more.

      Cheers,
      Daniel

       
  • Pavan says:

    Hi Daniel,

    I’m glad that I came to know about pg_activity by your post. But this is not addressing my database pain area.

    I’m using a PostgreSQL 9.1 database version with default checkpoint params. And the wal_level as ‘archive’.

    But to my wonder the archive logs are generating in a huge number, say 2000 files a day(31.25GB) from this Jan 1’st 2018.

    My main concern is to find all the DML’s which are responsible for this archive log generation, as I am running with serious disk space outage.

    How to get all the DML’s who are the culprits for this disk space problem. In other words, kindly help me to resolve this problem…

    Thanks in Advance.

    Cheers,
    Pavan

     
    • Daniel Westermann says:

      Hi Pavan,

      you can set log_statements=’DML’ to have all the DML statements logged to the logfile. Another way is to look at pg_stat_statements.

      Cheers,
      Daniel

       
      • Pavan says:

        Thank you so much Daniel. Kudoes

         
      • Pavan says:

        Hi Daniel,
        Yes I found many DML’s that are being run against the database. Now what should be the best plan of action to reduce huge WAL archiving. Really it is causing a serious disk problem nearly 1.35GB archive files are being generated per hour and I’m unable to investigate further in a better way.

        Please let me know if there is any better approach. Thanks in Advance.

        Regards,
        Pavan

         
  • Pavan says:

    Hi Daniel,

    I’m glad that I came to know about pg_activity by your post. But this is not addressing my database pain area.

    I’m using a PostgreSQL 9.1 database version with default checkpoint params. And the wal_level as ‘archive’.

    But to my wonder the archive logs are generating in a huge number, say 2000 files a day(31.25GB) from this Jan 1’st 2018.

    My main concern is to find all the DML’s which are responsible for this archive log generation, as I am running with serious disk space outage.

    How to get all the DML’s who are the culprits for this disk space problem. In other words, kindly help me to resolve this problem…

    Thanks in Advance.

    Cheers,
    Pavan

     

Leave a Reply


2 − two =

Daniel Westermann
Daniel Westermann

Senior Consultant and Technology Leader Open Infrastructure