In the last few months, we already did some tests with the Content Server 7.3, D2 4.7 and xPlore 1.6. As often, we found some funny/interesting behaviors and we faced some issues. In this blog I will talk about one issue I faced when I tried to install a new IndexAgent with xPlore 1.6. The Content Server used was a CS 7.3 (no patch available when I did this installation). This issue was present in both Silent and non-Silent installations.
So what was the issue? While trying to install a first IndexAgent, it looked like the installer wasn’t able to contact the Content Server. The GUI installer was stuck on the “Connection Broker Information”. After filling the information and clicking on “Next”, the Full Text Installer was stuck and nothing happened anymore.
Basically on this step of the installer, it is trying to connect to the docbroker using the information provided (docbroker hostname and port). Of course the docbroker and the docbase were running and responding properly.
To verify what was happening, I just checked the xPlore processes:
[xplore@full_text_server_01 ~]$ ps -ef | grep xplore xplore 2423 1 0 08:04 ? 00:00:00 /bin/sh ./startPrimaryDsearch.sh xplore 2425 2423 0 08:04 ? 00:00:00 /bin/sh $XPLORE_HOME/wildfly9.0.1/bin/standalone.sh xplore 2572 2425 5 08:04 ? 00:03:01 $XPLORE_HOME/java64/1.8.0_77/bin/java -D[Standalone] -server -XX:+UseCompressedOops -server -XX:+UseCompressedOops -XX:+UseCodeCacheFlushing -XX:ReservedCodeCacheSize=196m -Xms4096m -Xmx4096m -XX:MaxMetaspaceSize=512m -Xss1024k -XX:+UseCompressedOops -XX:+DisableExplicitGC -XX:-ReduceInitialCardMarks -Djava.awt.headless=true -Djboss.server.base.dir=$XPLORE_HOME/wildfly9.0.1/server/DctmServer_PrimaryDsearch -Djava.net.preferIPv4Stack=true -Dorg.jboss.boot.log.file=$XPLORE_HOME/wildfly9.0.1/server/DctmServer_PrimaryDsearch/log/server.log -Dlogging.configuration=file:$XPLORE_HOME/wildfly9.0.1/server/DctmServer_PrimaryDsearch/configuration/logging.properties -jar $XPLORE_HOME/wildfly9.0.1/jboss-modules.jar -mp $XPLORE_HOME/wildfly9.0.1/modules org.jboss.as.standalone -Djboss.home.dir=$XPLORE_HOME/wildfly9.0.1 -Djboss.server.base.dir=$XPLORE_HOME/wildfly9.0.1/server/DctmServer_PrimaryDsearch xplore 3977 2572 0 08:06 ? 00:00:00 $XPLORE_HOME/dsearch/cps/cps_daemon/bin/CPSDaemon $XPLORE_HOME/dsearch/cps/cps_daemon/PrimaryDsearch_local_configuration.xml Daemon0 9322 NORMAL xplore 4069 2572 0 08:06 ? 00:00:00 $XPLORE_HOME/dsearch/cps/cps_daemon/bin/CPSDaemon $XPLORE_HOME/dsearch/cps/cps_daemon/PrimaryDsearch_local_configuration.xml QDaemon0 9323 QUERY xplore 9591 9568 0 08:45 pts/3 00:00:00 /bin/sh ./configIndexagent.sh xplore 9592 9591 0 08:45 pts/3 00:00:08 $XPLORE_HOME/java64/1.8.0_77/jre/bin/java -Xmx262114000 -Xms262114000 com.zerog.lax.LAX /tmp/install.dir.9592/temp.lax /tmp/env.properties.9592 "-f" "config.properties" xplore 11099 9592 0 08:51 pts/3 00:00:02 $XPLORE_HOME/java64/1.8.0_77/jre/bin/java -cp /tmp/342186.tmp/testDocbrokerUtil.jar:$XPLORE_HOME/dfc/dfc.jar -Ddfc.properties.file=/tmp/install.dir.9592/dfc.properties com.documentum.install.shared.util.TestDocbrokerUtil content_server_01 1489 xplore 11928 9515 0 08:59 pts/1 00:00:00 ps -ef xplore 11929 9515 0 08:59 pts/1 00:00:00 grep xplore
As you can see above, the PrimaryDsearch is up & running so that’s good but there are two additional interesting processes:
xplore 9592 9591 0 08:45 pts/3 00:00:08 $XPLORE_HOME/java64/1.8.0_77/jre/bin/java -Xmx262114000 -Xms262114000 com.zerog.lax.LAX /tmp/install.dir.9592/temp.lax /tmp/env.properties.9592 "-f" "config.properties" xplore 11099 9592 0 08:51 pts/3 00:00:02 $XPLORE_HOME/java64/1.8.0_77/jre/bin/java -cp /tmp/342186.tmp/testDocbrokerUtil.jar:$XPLORE_HOME/dfc/dfc.jar -Ddfc.properties.file=/tmp/install.dir.9592/dfc.properties com.documentum.install.shared.util.TestDocbrokerUtil content_server_01 1489
The most interesting one is the second line: this is the command that the installer is starting to check that the docbroker is responding properly and this process is stuck. To do some testing, I just put the utilities and the needed elements to a test folder so I would be able to execute this command myself whenever I wanted. Then I killed the installer since it was stuck and I tried myself to contact the docbroker:
[xplore@full_text_server_01 ~]$ time $XPLORE_HOME/java64/1.8.0_77/jre/bin/java -cp /tmp/my.tmp/testDocbrokerUtil.jar:$XPLORE_HOME/dfc/dfc.jar -Ddfc.properties.file=/tmp/my.tmp/dfc.properties com.documentum.install.shared.util.TestDocbrokerUtil content_server_01 1489 ^C real 7m43.669s user 0m1.618s sys 0m0.153s
As you can see above, after 7min and 43 seconds, I just killed the process (^C) because it still didn’t return anything. Just to be sure, I checked our monitoring tool and it was able to connect to the docbase, remotely, without issue so the issue here was link somehow to the Full Text Server.
To go deeper, the only thing I could do was to perform some thread dumps of the java process to try to see where the bottleneck was. After doing that, it actually became quite clear that the server had an issue with the entropy generation because the threads were holding and waiting for the rsa libraries to get a random seed from the system. If you are running Documentum (or any other thing) on a Virtual Machine you will most probably want to change the default random generation because the entropy isn’t sufficient usually on VMs. We are always doing that for all our customer’s environments but since this was just a sandbox server to play with the newly released xPlore 1.6, this wasn’t setup at that time. With xPlore 1.5, this wouldn’t have caused any issue but it looks like xPlore 1.6 is much more demanding on the random seed generations. This has most probably something to do with some security improvements done in xPlore 1.6.
So to solve this issue, you just need to increase the entropy of your server. This can be done as follow (as root):
- Install the rng tools package
- Update the options in the /etc/sysconfig/rngd file
- Setup the service to start automatically at OS boot
You can basically use the following commands for that (I’m using a RHEL so it’s yum based):
[root@full_text_server_01 ~]# yum -y install rng-tools.x86_64 Loaded plugins: product-id, search-disabled-repos, security, subscription-manager Setting up Install Process Resolving Dependencies --> Running transaction check ... Transaction Test Succeeded Running Transaction Installing : rng-tools-5-2.el6_7.x86_64 1/1 Verifying : rng-tools-5-2.el6_7.x86_64 1/1 Installed: rng-tools.x86_64 0:5-2.el6_7 Complete! [root@full_text_server_01 ~]# rpm -qf /etc/sysconfig/rngd rng-tools-5-2.el6_7.x86_64 [root@full_text_server_01 ~]# [root@full_text_server_01 ~]# sed -i 's,EXTRAOPTIONS=.*,EXTRAOPTIONS=\"-r /dev/urandom -o /dev/random -t 0.1\",' /etc/sysconfig/rngd [root@full_text_server_01 ~]# cat /etc/sysconfig/rngd # Add extra options here EXTRAOPTIONS="-r /dev/urandom -o /dev/random -t 0.1" [root@full_text_server_01 ~]# [root@full_text_server_01 ~]# chkconfig --level 345 rngd on [root@full_text_server_01 ~]# chkconfig --list | grep rngd rngd 0:off 1:off 2:off 3:on 4:on 5:on 6:off [root@full_text_server_01 ~]# [root@full_text_server_01 ~]# service rngd start Starting rngd: [ OK ] [root@full_text_server_01 ~]#
When this is done, you need to force xPlore to use this new random generation mechanism. For that, just put in the profile of the xplore user one environment variable (as xplore):
[xplore@full_text_server_01 ~]$ echo 'export DEVRANDOM=/dev/urandom' >> ~/.bash_profile
When this has been done, you can just re-execute the java command and in something like 1 second the command should return a result like that:
[xplore@full_text_server_01 ~]$ $XPLORE_HOME/java64/1.8.0_77/jre/bin/java -cp /tmp/my.tmp/testDocbrokerUtil.jar:$XPLORE_HOME/dfc/dfc.jar -Ddfc.properties.file=/tmp/my.tmp/dfc.properties com.documentum.install.shared.util.TestDocbrokerUtil content_server_01 1489 0 [main] ERROR com.documentum.fc.common.impl.logging.LoggingConfigurator - Problem locating log4j configuration 1 [main] WARN com.documentum.fc.common.impl.logging.LoggingConfigurator - Using default log4j configuration [xplore@full_text_server_01 ~]$
These two messages aren’t important, they are just printed because there is no log4j properties file in the command line so it is using the default one. The only important thing here is that the command returns something so it is working! To confirm that, you can just run the installer once more and this time the screen shouldn’t be stuck on the Connection Broker Information.