Documentum – A first experience with the Life Sciences Suite 16.6.1 silent installation

A few months ago, I was working on automating an LSS installation for a customer so I thought about sharing my experience around that topic… LSS is the Life Sciences product from OpenText (formerly EMC->Dell->OpenText). It can be split into four sub-products, which can be installed independently or all together, depending on the needs. These four products are the LSTMF (eTrial Master File Solution), LSQM (Quality and Manufacturing Solution), LSRD (Research and Development Solution) and finally the LSSSV (Submission Store & View Solution). When they are all bundled together, they form the Life Sciences Suite. In this blog, I will talk about the LSS product globally.

The LSS installation can be done either manually or silently. Contrary to most OpenText products which uses installers (.exe, .bin) to provide either a GUI or silent processing, the silent installation of the LSS is just a shell script or to be exact some shell scripts that you will need to execute one by one (depending on what you need exactly). In this case, my task was to automate the LSS installation to be able to integrate that into Kubernetes using Ansible playbooks. Therefore, OpenText provides some shell scripts to silently install the LSS but there is still some preparation work to be done before and some verifications after. The content of the LSS package looks like that:

[dmadmin@cspg-0 ~]$ workspace="/tmp/lss/"
[dmadmin@cspg-0 ~]$ package="LSSuite_16.6.1000.0126.tar"
[dmadmin@cspg-0 ~]$
[dmadmin@cspg-0 ~]$ mkdir -p ${workspace}
[dmadmin@cspg-0 ~]$ cd ${workspace}
[dmadmin@cspg-0 lss]$ cp /tmp/${package} ./
[dmadmin@cspg-0 lss]$ tar -xf ${package}
[dmadmin@cspg-0 lss]$
[dmadmin@cspg-0 lss]$ cd ./*/
[dmadmin@cspg-0 LSSuite]$ ls -l
total 216
drwxr-x--- 3 dmadmin dmadmin   152 Dec  2 17:09 CDFD2Plugins
drwxr-x--- 2 dmadmin dmadmin   152 Dec  2 17:09 ControlledPrintPluginJars
-rw-r----- 1 dmadmin dmadmin 13479 Oct  4 09:15 LSConfigImport.sh
drwxr-x--- 2 dmadmin dmadmin   152 Dec  2 17:09 SDK
drwxr-x--- 3 dmadmin dmadmin   152 Oct 21 09:52 Util
drwxr-x--- 2 dmadmin dmadmin   152 Dec  2 17:09 config
drwxr-x--- 2 dmadmin dmadmin  8192 Dec  2 17:09 config_exports
-rw-r----- 1 dmadmin dmadmin   544 Oct  4 09:15 copyright.txt
drwxr-x--- 2 dmadmin dmadmin  8192 Dec  2 17:09 dars
-rw-r----- 1 dmadmin dmadmin  4231 Oct  4 09:15 iHubConfigImport.sh
drwxr-x--- 2 dmadmin dmadmin  8192 Dec  2 17:09 iHubDependentJar
drwxr-x--- 2 dmadmin dmadmin   152 Dec  2 17:09 iHubJDBCConnector
drwxr-x--- 2 dmadmin dmadmin   152 Dec  2 17:09 iHubReports
-rw-r----- 1 dmadmin dmadmin   294 Oct  4 09:15 ihubsolution.properties
-rw-r----- 1 dmadmin dmadmin  8467 Oct  4 09:15 install.properties
-rw-r----- 1 dmadmin dmadmin  8384 Oct  4 09:15 install.sh
-rw-r----- 1 dmadmin dmadmin 28838 Oct  4 09:15 install.xml
drwxr-x--- 2 dmadmin dmadmin  8192 Dec  2 17:09 javadocs
drwxr-x--- 2 dmadmin dmadmin  8192 Dec  2 17:09 lib
drwxr-x--- 2 dmadmin dmadmin  8192 Dec  2 17:09 method_server_lib
-rw-r----- 1 dmadmin dmadmin  3096 Oct  4 09:15 myInsightPostInstall.sh
-rw-r----- 1 dmadmin dmadmin   224 Oct  4 09:15 nodmadmin.installparam.xml
-rw-r----- 1 dmadmin dmadmin   981 Oct  4 09:15 readme.txt
drwxr-x--- 3 dmadmin dmadmin  8192 Dec  2 17:09 scripts
drwxr-x--- 2 dmadmin dmadmin   152 Dec  2 17:09 sharepointconnector
-rw-r----- 1 dmadmin dmadmin  1779 Oct  4 09:15 solution.properties
drwxr-x--- 2 dmadmin dmadmin  8192 Dec  2 17:09 updateVersion
-rw-r----- 1 dmadmin dmadmin    69 Oct  4 09:15 version.txt
drwxr-x--- 2 dmadmin dmadmin  8192 Dec  2 17:09 war
[dmadmin@cspg-0 LSSuite]$

Above, you have all the files/folders that come with the LSS package. There are several “.sh” scripts, that’s the shell scripts for the silent installation and they are using parameters/variables that have been defined in the different “.properties” files. What changes between the different LSS products – in addition to the DARs/JARs – is the value of the variables in these properties files.

In this blog, there are a few important things that I wanted to highlight:

1. You shouldn’t blindly trust what OpenText is providing as variables’ values. I have seen, several times, wrong values in properties files because of copy/paste from previous versions or worse, copy/paste from other LSS products. Therefore, before installing a new version/product, the first thing that I usually do is checking and validating that all variables have been set properly according to what I expect the soft to do. This obviously need some knowledge on how the LSS is behaving based on the parameters’ value. If you have some customization or some specific requirements, it’s also the moment to add/update the properties files to match your needs.
2. Before installing, you should really check carefully the release note. This is obvious for most people, but I know it can be quite painful to do for all products. Because the LSS “installer” is actually just shell script, it can very easily contain mistakes or typos, contrary to most Documentum binaries (or not!). So check the release note of the LSS because there are often known issues and bugs in the released products, even inside the shell scripts and variables’ values… Therefore, taking a few minutes for that will be worth it, that’s for sure, to make sure there is no known bugs with the silent scripts/variables before running them.
3. OpenText doesn’t always manage the return/exit codes (to not say never). If you don’t know what it is, return/exit codes can be seen as a numerical representation of the result of a command. In UNIX for example, each command that you will execute will return an exit code which can be checked. A value of 0 usually means that the command completed successfully while a value from 1 to 255 means the opposite, for different reasons (a value of 2 means a different issue compared to a value of 1). While scripting, that’s usually a very good and efficient (the best?) way to make sure everything is going as planned. The issue here is that OpenText is using the return codes at some places but clearly not everywhere… Therefore, you have maybe something like 75% of the scripting that can fail, and the installation will still continue anyway. So, what to do with that? Well, either you update the scripts provided by OpenText or you make sure to check thoroughly the log files generated by the installation. Updating the scripts isn’t really a good idea since they will be overwritten for each patch, so it only leaves you with not so much of a choice… 🙂

Let’s talk a little bit more about this third point. The simplest way to demonstrate that is to purposely try to execute the shell scripts with missing parameters and check the return code:

[dmadmin@cspg-0 LSSuite]$ cat version.txt; echo
OpenText Life Sciences Solution Suite
Build Version = 16.6.1000.0126
[dmadmin@cspg-0 LSSuite]$
[dmadmin@cspg-0 LSSuite]$ export LS_DOCKER_INSTALL=true
[dmadmin@cspg-0 LSSuite]$ ./install.sh
Installation started on
12022019
Please make sure the JMS is stopped before running the script
./install.sh: line 38: [: =: unary operator expected
./install.sh: line 38: [: =: unary operator expected
./install.sh: line 41: [: =: unary operator expected
./install.sh: line 41: [: =: unary operator expected
Please provide the supported Content Server version [7.1, 7.2, 7.3, 16.4].
[dmadmin@cspg-0 LSSuite]$
[dmadmin@cspg-0 LSSuite]$ echo $?
1
[dmadmin@cspg-0 LSSuite]$
[dmadmin@cspg-0 LSSuite]$
[dmadmin@cspg-0 LSSuite]$ ./LSConfigImport.sh
Installation started on
12022019
Please provide the supported Content Server version
[dmadmin@cspg-0 LSSuite]$
[dmadmin@cspg-0 LSSuite]$ echo $?
0
[dmadmin@cspg-0 LSSuite]$

As you can see above, to silently install the LSS, you will need to set the variable “LS_DOCKER_INSTALL” to true first, otherwise it will prompt you to fill some variables’ value. If you are trying to execute the first shell script (install.sh), then it will fail because there are missing parameters (it expects 4 parameters) and the return code is 1, meaning that something went wrong: that’s the expected result so this is good! Doing the same thing for the second shell script (LSConfigImport.sh) will also fail because of missing parameters (it expects 14 parameters) but this time, the return code is 0, meaning that there were “no errors”… However, nothing has been done since it actually failed. How are you supposed to detect that the installation failed in this case? Well, that’s a good question! LSS will generate some logs that will be put in the working directory so a “solution” can be to make sure that all the expected logs are present and that their content is also “correct”… That’s not really a piece of cake but is there any other choice?

I have been installing LSS dozens of times and I saw several times errors reported on the logs that weren’t displayed on the screen during the execution of the scripts. And vice versa! Here is an example:

[dmadmin@cspg-0 LSSuite]$ cd ./working/logs/
[dmadmin@cspg-0 logs]$
[dmadmin@cspg-0 logs]$ grep -i _E_ *
LSSuite_postInstall.log:[DM_SYSOBJECT_E_CANT_ACCESS_FILE]error:  "Cannot access file 'jp-regional_1-0-normalize.xslt' due to Operating System error."
LSSuite_postInstall.log:[DM_API_E_CANT_UPDATE]error:  "Cannot Save/Check-in object <> because the object is marked invalid.  This is most likely due to an error from a prior operation performed on the object."
LSSuite_postInstall.log:[DM_SYSOBJECT_E_CANT_ACCESS_FILE]error:  "Cannot access file 'stf-2-2-normalize.xslt' due to Operating System error."
LSSuite_postInstall.log:[DM_API_E_CANT_UPDATE]error:  "Cannot Save/Check-in object <> because the object is marked invalid.  This is most likely due to an error from a prior operation performed on the object."
LSSuite_postInstall.log:[DM_SYSOBJECT_E_CANT_ACCESS_FILE]error:  "Cannot access file 'us-regional-2-3-normalize.xslt' due to Operating System error."
LSSuite_postInstall.log:[DM_API_E_CANT_UPDATE]error:  "Cannot Save/Check-in object <> because the object is marked invalid.  This is most likely due to an error from a prior operation performed on the object."
[dmadmin@cspg-0 logs]$

The above errors for example are all linked to the same iAPI script but just looking at these messages, it’s not always that easy to find out where the issue is coming from. Most of the log files are just an aggregation of iAPI/iDQL session logging without any evidences of which script is being executed or what’s its purpose. For the above errors, finding the correct iAPI script was pretty easy because the messages contain three file names. Therefore, with just a simple grep command you can find that there is an iAPI script named “addRenditions.api” that seems to be the cause of these errors. I have seen cases where the errors were too generic and therefore, I had to count the number of iAPI/iDQL logins in the log (number of message “Connected to OpenText Documentum Server running…”). Then list all the iAPI/iDQL scripts that are being executed based on the properties files. Then loop through all these iAPI/iDQL scripts and count again the number of logins to find which command/script generated the errors… Easy, right?

Anyway, this particular issue is documented in the installation guide of the LSS. For some reasons, OpenText created an iAPI script and used a relative path for the files but it’s not correct because of how the LSS executes it… The solution is therefore to update the command to use the full path of the file:

[dmadmin@cspg-0 logs]$ cd ../../
[dmadmin@cspg-0 LSSuite]$
[dmadmin@cspg-0 LSSuite]$ cat ./scripts/scripts/addRenditions.api
retrieve,c,cd_ectd_xml_schema where object_name='jp-regional_1-0'
addrendition,c,l,jp-regional_1-0-normalize.xslt,xsl_ns
save,c,l

retrieve,c,cd_ectd_xml_schema where object_name='stf_2-2'
addrendition,c,l,stf-2-2-normalize.xslt,xsl_ns
save,c,l

retrieve,c,cd_ectd_xml_schema where object_name='us-regional_2-3'
addrendition,c,l,us-regional-2-3-normalize.xslt,xsl_ns
save,c,l
[dmadmin@cspg-0 LSSuite]$
[dmadmin@cspg-0 LSSuite]$ ls -l $(pwd)/scripts/scripts/stf-2-2-normalize.xslt
-rw-r----- 1 dmadmin dmadmin 3748 Oct  4 08:48 /tmp/lss/LSSuite/scripts/scripts/stf-2-2-normalize.xslt
[dmadmin@cspg-0 LSSuite]$
[dmadmin@cspg-0 LSSuite]$ iapi REPO1 -Udmadmin -Pxxx -r$(pwd)/scripts/scripts/addRenditions.api

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0170.0080

Connecting to Server using docbase REPO1
[DM_SESSION_I_SESSION_START]info:  "Session 010f12345001568c started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0170.0234  Linux64.Postgres
Session id is s0
API> ...
080f123450005514
API> ...
[DM_SYSOBJECT_E_CANT_ACCESS_FILE]error:  "Cannot access file 'jp-regional_1-0-normalize.xslt' due to Operating System error."

API> ...
[DM_API_E_CANT_UPDATE]error:  "Cannot Save/Check-in object <> because the object is marked invalid.  This is most likely due to an error from a prior operation performed on the object."

API> ...
080f123450005518
API> ...
[DM_SYSOBJECT_E_CANT_ACCESS_FILE]error:  "Cannot access file 'stf-2-2-normalize.xslt' due to Operating System error."

API> ...
[DM_API_E_CANT_UPDATE]error:  "Cannot Save/Check-in object <> because the object is marked invalid.  This is most likely due to an error from a prior operation performed on the object."

API> ...
080f12345000551e
API> ...
[DM_SYSOBJECT_E_CANT_ACCESS_FILE]error:  "Cannot access file 'us-regional-2-3-normalize.xslt' due to Operating System error."

API> ...
[DM_API_E_CANT_UPDATE]error:  "Cannot Save/Check-in object <> because the object is marked invalid.  This is most likely due to an error from a prior operation performed on the object."

API> Bye
[dmadmin@cspg-0 LSSuite]$
[dmadmin@cspg-0 LSSuite]$ sed -i "s@^addrendition,c,l,([a-zA-Z])@addrendition,c,l,$(pwd)/scripts/scripts/1@" ./scripts/scripts/addRenditions.api
[dmadmin@cspg-0 LSSuite]$
[dmadmin@cspg-0 LSSuite]$ cat ./scripts/scripts/addRenditions.api
retrieve,c,cd_ectd_xml_schema where object_name='jp-regional_1-0'
addrendition,c,l,/tmp/lss/LSSuite/scripts/scripts/jp-regional_1-0-normalize.xslt,xsl_ns
save,c,l

retrieve,c,cd_ectd_xml_schema where object_name='stf_2-2'
addrendition,c,l,/tmp/lss/LSSuite/scripts/scripts/stf-2-2-normalize.xslt,xsl_ns
save,c,l

retrieve,c,cd_ectd_xml_schema where object_name='us-regional_2-3'
addrendition,c,l,/tmp/lss/LSSuite/scripts/scripts/us-regional-2-3-normalize.xslt,xsl_ns
save,c,l
[dmadmin@cspg-0 LSSuite]$
[dmadmin@cspg-0 LSSuite]$ iapi REPO1 -Udmadmin -Pxxx -r$(pwd)/scripts/scripts/addRenditions.api

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0170.0080

Connecting to Server using docbase REPO1
[DM_SESSION_I_SESSION_START]info:  "Session 010f12345001568e started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0170.0234  Linux64.Postgres
Session id is s0
API> ...
080f123450005514
API> ...
OK
API> ...
OK
API> ...
080f123450005518
API> ...
OK
API> ...
OK
API> ...
080f12345000551e
API> ...
OK
API> ...
OK
API> Bye
[dmadmin@cspg-0 LSSuite]$

My experience with the OpenText Life Sciences product(s) in the past few years is that it’s always an adventure… It might be fun or interesting but before all, it’s most probably going to be a little bit challenging ;).