Infrastructure at your Service

Morgan Patou

Documentum – dmqdocbroker/iapi/idql not working because of dbor.properties.lck

Have you ever faced an issue where dmqdocbroker, iapi, idql and the likes aren’t able to communicate at all with any Docbroker (connection broker)? Here, I’m not talking about potentially wrong hostname, port or connect modes, which might prevent you to reach a Docbroker if it’s not configured properly because this will still most likely reply to you with an error message… I’m really talking about the utility/binaries that cannot communicate anymore, it’s like all messages are sent to the void and nothing will ever respond (is that a black-hole I’m seeing?)!

Earlier this month, I suddenly had this behavior at one of our customer on two out of dozens of Documentum Servers. Everything seemed to be up&running, all the processes were there:

[[email protected] ~]$ ps -ef
UID      PID PPID C  STIME TTY       TIME CMD
dmadmin 7005    1 0  14:11 ?     00:00:00 ./dmdocbroker -port 1489 -init_file $DOCUMENTUM/dba/Docbroker.ini
dmadmin 7014    1 0  14:11 ?     00:00:00 ./dmdocbroker -port 1487 -init_file $DOCUMENTUM/dba/DocbrokerExt.ini
dmadmin 7077    1 0  14:11 ?     00:00:07 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini
dmadmin 7087    1 0  14:11 ?     00:00:07 ./documentum -docbase_name REPO1 -security acl -init_file $DOCUMENTUM/dba/config/REPO1/server.ini
dmadmin 7100 7077 0  14:11 ?     00:00:00 $DM_HOME/bin/mthdsvr master 0xfd7308a8, 0x7f9f93d81000, 0x223000 1000726 5 7077 GR_REPO $DOCUMENTUM/dba/log
dmadmin 7101 7100 0  14:11 ?     00:00:04 $DM_HOME/bin/mthdsvr worker 0xfd7308a8, 0x7f9f93d81000, 0x223000 1000726 5 0 GR_REPO $DOCUMENTUM/dba/log
dmadmin 7102 7087 0  14:11 ?     00:00:00 $DM_HOME/bin/mthdsvr master 0xfd7308be, 0x7fe2fe3ac000, 0x223000 1000727 5 7087 REPO1 $DOCUMENTUM/dba/log
dmadmin 7121 7102 0  14:11 ?     00:00:03 $DM_HOME/bin/mthdsvr worker 0xfd7308be, 0x7fe2fe3ac000, 0x223000 1000727 5 0 REPO1 $DOCUMENTUM/dba/log
dmadmin 7122 7100 0  14:11 ?     00:00:03 $DM_HOME/bin/mthdsvr worker 0xfd7308a8, 0x7f9f93d81000, 0x223000 1000726 5 1 GR_REPO $DOCUMENTUM/dba/log
dmadmin 7123 7077 0  14:11 ?     00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini
dmadmin 7124 7077 0  14:11 ?     00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini
dmadmin 7144 7102 0  14:11 ?     00:00:04 $DM_HOME/bin/mthdsvr worker 0xfd7308be, 0x7fe2fe3ac000, 0x223000 1000727 5 1 REPO1 $DOCUMENTUM/dba/log
dmadmin 7148 7087 0  14:11 ?     00:00:00 ./documentum -docbase_name REPO1 -security acl -init_file $DOCUMENTUM/dba/config/REPO1/server.ini
dmadmin 7149 7087 0  14:11 ?     00:00:00 ./documentum -docbase_name REPO1 -security acl -init_file $DOCUMENTUM/dba/config/REPO1/server.ini
dmadmin 7165 7100 0  14:11 ?     00:00:04 $DM_HOME/bin/mthdsvr worker 0xfd7308a8, 0x7f9f93d81000, 0x223000 1000726 5 2 GR_REPO $DOCUMENTUM/dba/log
dmadmin 7166 7077 0  14:11 ?     00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini
dmadmin 7167 7102 0  14:11 ?     00:00:03 $DM_HOME/bin/mthdsvr worker 0xfd7308be, 0x7fe2fe3ac000, 0x223000 1000727 5 2 REPO1 $DOCUMENTUM/dba/log
dmadmin 7168 7087 0  14:11 ?     00:00:00 ./documentum -docbase_name REPO1 -security acl -init_file $DOCUMENTUM/dba/config/REPO1/server.ini
dmadmin 7169 7100 0  14:11 ?     00:00:04 $DM_HOME/bin/mthdsvr worker 0xfd7308a8, 0x7f9f93d81000, 0x223000 1000726 5 3 GR_REPO $DOCUMENTUM/dba/log
dmadmin 7187 7077 0  14:11 ?     00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini
dmadmin 7190 7102 0  14:11 ?     00:00:03 $DM_HOME/bin/mthdsvr worker 0xfd7308be, 0x7fe2fe3ac000, 0x223000 1000727 5 3 REPO1 $DOCUMENTUM/dba/log
dmadmin 7194 7087 0  14:11 ?     00:00:00 ./documentum -docbase_name REPO1 -security acl -init_file $DOCUMENTUM/dba/config/REPO1/server.ini
dmadmin 7210 7100 0  14:11 ?     00:00:03 $DM_HOME/bin/mthdsvr worker 0xfd7308a8, 0x7f9f93d81000, 0x223000 1000726 5 4 GR_REPO $DOCUMENTUM/dba/log
dmadmin 7213 7077 0  14:11 ?     00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini
dmadmin 7215 7102 0  14:11 ?     00:00:04 $DM_HOME/bin/mthdsvr worker 0xfd7308be, 0x7fe2fe3ac000, 0x223000 1000727 5 4 REPO1 $DOCUMENTUM/dba/log
dmadmin 7225 7087 0  14:11 ?     00:00:00 ./documentum -docbase_name REPO1 -security acl -init_file $DOCUMENTUM/dba/config/REPO1/server.ini
dmadmin 7334    1 0  14:12 ?     00:00:00 /bin/sh $JMS_HOME/server/startMethodServer.sh
dmadmin 7336 7334 0  14:12 ?     00:00:00 /bin/sh $JMS_HOME/bin/standalone.sh
dmadmin 7447 7336 21 14:12 ?     00:02:57 $JAVA_HOME/bin/java -D[Standalone] -server -XX:+UseCompressedOops -server -XX:+UseCompressedOops -Xms8g -Xmx8g -XX:MaxMetaspaceSize=512m -XX
dmadmin 7695 7077 0  14:12 ?     00:00:04 ./dm_agent_exec -enable_ha_setup 1 -docbase_name GR_REPO.GR_REPO -docbase_owner dmadmin -sleep_duration 0
dmadmin 7698 7087 0  14:12 ?     00:00:04 ./dm_agent_exec -enable_ha_setup 1 -docbase_name REPO1.REPO1 -docbase_owner dmadmin -sleep_duration 0
dmadmin 7908 7077 0  14:13 ?     00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini
dmadmin 7918 7087 0  14:13 ?     00:00:00 ./documentum -docbase_name REPO1 -security acl -init_file $DOCUMENTUM/dba/config/REPO1/server.ini
dmadmin 8269 7077 0  14:21 ?     00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini
dmadmin 8270 7077 0  14:21 ?     00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini
dmadmin 8327 6370 0  14:27 pts/1 00:00:00 ps -ef
[[email protected] ~]$

 

However, I could see the communication issues by looking at the Repository log because it would show that the AgentExec was actually not connected, even after almost 20 minutes:

[[email protected] ~]$ cd $DOCUMENTUM/dba/log/REPO1/agentexec/
[[email protected] agentexec]$ date
Mon Apr 12 14:29:03 UTC 2021
[[email protected] agentexec]$
[[email protected] agentexec]$ tail -8 ../../REPO1.log
2021-04-12T14:11:56.453407      7087[7087]      0000000000000000        [DM_WORKFLOW_I_AGENT_START]info:  "Workflow agent worker (pid : 7194, session 010f12345000000c) is started sucessfully."
2021-04-12T14:11:57.455899      7087[7087]      0000000000000000        [DM_SERVER_I_START]info:  "Sending Initial Docbroker check-point "

2021-04-12T14:11:57.547764      7087[7087]      0000000000000000        [DM_MQ_I_DAEMON_START]info:  "Message queue daemon (pid : 7225, session 010f123450000456) is started sucessfully."
2021-04-12T14:11:58.348442      7223[7223]      010f123450000003        [DM_DOCBROKER_I_PROJECTING]info:  "Sending information to Docbroker located on host (cs-0.domain.com) with port (1490).  Information: (Config(REPO1), Proximity(1), Status(Open), Dormancy Status(Active))."
2021-04-12T14:11:58.661666      7223[7223]      010f123450000003        [DM_DOCBROKER_I_PROJECTING]info:  "Sending information to Docbroker located on host (cs-0.domain.com) with port (1488).  Information: (Config(REPO1), Proximity(1), Status(Open), Dormancy Status(Active))."
2021-04-12T14:11:58.959490      7223[7223]      010f123450000003        [DM_DOCBROKER_I_PROJECTING]info:  "Sending information to Docbroker located on host (cs-1.domain.com) with port (1490).  Information: (Config(REPO1), Proximity(2), Status(Open), Dormancy Status(Active))."
Mon Apr 12 14:12:55 2021 [INFORMATION] [AGENTEXEC 7698] Detected during program initialization: Version: 16.4.0200.0256  Linux64
[[email protected] agentexec]$
[[email protected] agentexec]$ # Previous startup from the AgentExec logs showing that it didn't start yet
[[email protected] agentexec]$ tail -2 agentexec.log
Sat Apr 10 19:20:30 2021 [INFORMATION] [LAUNCHER 23135] Detected during program initialization: Version: 16.4.0200.0256 Linux64
Sun Apr 11 19:20:26 2021 [INFORMATION] [LAUNCHER 2890] Detected during program initialization: Version: 16.4.0200.0256 Linux64
[[email protected] agentexec]$

 

The interesting part is that the Repositories have all been started properly and projected to the Docbroker. However, any client from the Documentum Server locally wouldn’t be able to connect to the Docbroker. Even more interesting, this was actually a HA environment with 2 CS. The Documentum Server hosting the Primary CS (I will call it cs-0) had the issue while the Documentum Server hosting the Remote CS (I will call it cs-1) had no problem. Executing the dmqdocbroker on the cs-0 to ping the Docbroker of the cs-0 never gave a response, however doing the exact same command to ping the Docbroker of the cs-0 but from the cs-1 host did work without any problem and showed the correct projection of the repositories:

## on cs-0 (Documentum Server hosting the PCS)
[[email protected] ~]$ hostname -f
cs-0.domain.com
[[email protected] ~]$
[[email protected] ~]$ time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping
^C
real 0m53.513s
user 0m6.132s
sys 0m0.578s
[[email protected] ~]$
[[email protected] ~]$ time echo quit | iapi REPO1.REPO1 -Udmadmin -Pxxx
^C
real 0m46.431s
user 0m6.241s
sys 0m0.575s
[[email protected] ~]$
[[email protected] ~]$ time echo quit | iapi REPO1.cs-1_REPO1 -Udmadmin -Pxxx
^C
real 0m35.694s
user 0m6.163s
sys 0m0.582s
[[email protected] ~]$

## on cs-1 (Documentum Server hosting the RCS)
[[email protected] ~]$ hostname -f
cs-1.domain.com
[[email protected] ~]$
[[email protected] ~]$ time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0200.0080
Using specified port: 1489
Successful reply from docbroker at host (cs-0) on port(1490) running software version (16.4.0200.0256 Linux64).

real 0m3.499s
user 0m6.950s
sys 0m0.469s
[[email protected] ~]$
[[email protected] ~]$ time echo quit | iapi REPO1.cs-1_REPO1 -Udmadmin -Pxxx

OpenText Documentum iapi - Interactive API interface
Copyright (c) 2018. OpenText Corporation
All rights reserved.
Client Library Release 16.4.0200.0080

Connecting to Server using docbase REPO1.cs-1_REPO1
[DM_SESSION_I_SESSION_START]info: "Session 010f1234501b635c started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0200.0256 Linux64.Oracle
Session id is s0
API> Bye

real 0m5.032s
user 0m7.401s
sys 0m0.487s
[[email protected] ~]$
[[email protected] ~]$ time echo quit | iapi REPO1.REPO1 -Udmadmin -Pxxx

OpenText Documentum iapi - Interactive API interface
Copyright (c) 2018. OpenText Corporation
All rights reserved.
Client Library Release 16.4.0200.0080

Connecting to Server using docbase REPO1.REPO1
[DM_SESSION_I_SESSION_START]info: "Session 010f1234501b6506 started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0200.0256 Linux64.Oracle
Session id is s0
API> Bye

real 0m5.315s
user 0m7.976s
sys 0m0.515s
[[email protected] ~]$

 

This shows that the issue isn’t the Docbroker or the Repositories themselves but rather the utility/binaries present on the cs-0 that cannot open communication channels with the local Docbroker, for some reasons… Even after setting debugging on the Docbroker, I could see communications when the dmqdocbroker utility was used on the cs-1 host but nothing was showing-up if the same command was used on the cs-0 host instead. You can enable some logs for the Docbroker by adding “trace=true” into the Docbroker.ini file and you can also add some other traces by setting the following environment variables (value can be 1 or 10 for example) and then restart the Docbroker: “export DM_DOCBROKER_TRACE=1; export DM_DEBUG_BROKER=1; export DM_TRANS_LOG=1“. Additionally, you can also add options to the launch script, just like for the Repository part: “-odocbroker_trace -onettrace_all_option -oxxx“.

Unfortunately, the dmqdocbroker utility uses the dmawk binary and the iapi/idql are also binaries so it’s rather difficult to debug further without the source code… After some testing/debugging, I found something rather hard to believe… All the binaries of the Documentum Server looked OK, they were no changes done in the past few weeks and the files were the same (same hash) than on the cs-1 for example. As you probably know, dmqdocbroker/iapi/idql will use the dfc.properties from the folder “$DOCUMENTUM_SHARED/config/” (with $DOCUMENTUM_SHARED=$DOCUMENTUM forced, starting in 16.4). Therefore, I have been looking into this folder for anything that might disrupt the proper behavior of the utility/binaries. All the files in this folder were 100% identical between cs-0 and cs-1, except for the encrypted password of the dm_bof_registry as well as the dfc.keystore since both of these are generated once. This would mean that the issue wouldn’t be there, but it was. I started looking into other areas to try to find the root cause but nothing was working. Then, I came back to the config folder and simply tried to empty it… Somehow, the dmqdocbroker was working again, magically! I mean, it printed many errors because the files log4j.properties, dfc.properties and dfc.keystore weren’t there but it replied something… What to do then? Well, I went step by step, putting back the files one by one, as they are supposed to be, and then executing the dmqdocbroker again to see if it stops working.

The files dfc.properties, log4j.properties, dfcfull.properties, dfc.keystore and all the cache folders were restored properly and the dmqdocbroker was still working without any problem… So what the hell? That’s more or less all of the files, isn’t it? True, that’s all the files, minus the dbor ones: dbor.properties and dbor.properties.lck. At this customer, these files are empty because no configuration was needed. It would be very hard to believe that this could be the issue, right? Well, have a look for yourself:

[[email protected] ~]$ cd $DOCUMENTUM_SHARED/config/
[[email protected] config]$ ls -l
total 140
drwxr-x--- 7 dmadmin dmadmin  4096 Jul 26 2020 ServerApps
drwxr-x--- 9 dmadmin dmadmin  4096 Jul 26 2020 Shared
drwxr-x--- 7 dmadmin dmadmin  4096 Jul 26 2020 acs
drwxr-x--- 7 dmadmin dmadmin  4096 Jul 26 2020 bpm
-rwxr-x--- 1 dmadmin dmadmin     0 Jul 22 2020 dbor.properties
-rw-rw-r-- 1 dmadmin dmadmin     0 Jul 26 2020 dbor.properties.lck
-rw-rw-r-- 1 dmadmin dmadmin  2152 Jul 26 2020 dfc.keystore
-rw-rw-r-- 1 dmadmin dmadmin   481 Jul 26 2020 dfc.properties
-rw-rw---- 1 dmadmin dmadmin    70 Jul 22 2020 dfc.properties.bak.0
-rwxr-x--- 1 dmadmin dmadmin   242 Jul 26 2020 dfc.properties.bak.1
-rw-rw-r-- 1 dmadmin dmadmin   271 Jul 26 2020 dfc.properties.bak.2
-rw-rw-r-- 1 dmadmin dmadmin   323 Jul 26 2020 dfc.properties.bak.3
-rw-rw-r-- 1 dmadmin dmadmin   481 Jul 26 2020 dfc.properties.bak.4
-rw-rw-r-- 1 dmadmin dmadmin   482 Jul 26 2020 dfc.properties.bak.5
-rwxrwx--- 1 dmadmin dmadmin 79268 Jul 22 2020 dfcfull.properties
-rwxr-x--- 1 dmadmin dmadmin  1242 Jul 26 2020 log4j.properties
[[email protected] config]$
[[email protected] config]$ # With the initial content, dmqdocbroker isn't working
[[email protected] config]$ date; time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping; date
Wed Apr 14 07:43:28 UTC 2021
^C
real    0m22.718s
user    0m6.401s
sys     0m0.853s
Wed Apr 14 07:43:51 UTC 2021
[[email protected] config]$
[[email protected] config]$ mkdir test
[[email protected] config]$ mv * test/
mv: cannot move 'test' to a subdirectory of itself, 'test/test'
[[email protected] config]$
[[email protected] config]$ ls -l
total 4
drwxr-x--- 6 dmadmin dmadmin 4096 Apr 14 07:44 test
[[email protected] config]$
[[email protected] config]$ # With the folder empty, dmqdocbroker is "working" (errors but expected ones)
[[email protected] config]$ date; time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping; date
Wed Apr 14 07:45:15 UTC 2021
0 [main] ERROR com.documentum.fc.common.impl.logging.LoggingConfigurator  - Problem locating log4j configuration
1 [main] WARN com.documentum.fc.common.impl.logging.LoggingConfigurator  - Using default log4j configuration
3 [main] ERROR com.documentum.fc.common.impl.preferences.PreferencesManager  - [DFC_PREFERENCE_LOAD_FAILED] Failed to load persistent preferences from null
java.io.FileNotFoundException: dfc.properties
        at com.documentum.fc.common.impl.preferences.PreferencesManager.locateMainPersistentStore(PreferencesManager.java:378)
        at com.documentum.fc.common.impl.preferences.PreferencesManager.readPersistentProperties(PreferencesManager.java:329)
        at com.documentum.fc.common.impl.preferences.PreferencesManager.<init>(PreferencesManager.java:37)
        ...
2862 [main] ERROR com.documentum.fc.client.security.impl.IdentityManager  - [DFC_SECURITY_IDENTITY_INIT] no identity initialization or incomplete identity initialization
DfException:: THREAD: main; MSG: ; ERRORCODE: ff; NEXT: null
        at com.documentum.fc.client.security.impl.JKSKeystoreUtil.creteNewKeystoreFile(JKSKeystoreUtil.java:425)
        at com.documentum.fc.client.security.impl.JKSKeystoreUtil.createNewKeystore(JKSKeystoreUtil.java:209)
        at com.documentum.fc.client.security.impl.DfcIdentityKeystore.applyDfcInitPolicy(DfcIdentityKeystore.java:95)
        ...
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0200.0080
Using specified port: 1489
Successful reply from docbroker at host (cs-0) on port(1490) running software version (16.4.0200.0256 Linux64).

real    0m3.763s
user    0m6.265s
sys     0m0.672s
Wed Apr 14 07:45:19 UTC 2021
[[email protected] config]$
[[email protected] config]$ ls -l
total 12
drwxr-x--- 8 dmadmin dmadmin 4096 Apr 14 07:45 documentum
-rw-r----- 1 dmadmin dmadmin 3245 Apr 14 07:45 log4j.log
drwxr-x--- 6 dmadmin dmadmin 4096 Apr 14 07:44 test
-rw-r----- 1 dmadmin dmadmin    0 Apr 14 07:45 trace.log
[[email protected] config]$
[[email protected] config]$ rm -rf documentum/ log4j.log  trace.log
[[email protected] config]$
[[email protected] config]$ mv test/log4j.properties ./
[[email protected] config]$ mv test/dfc.properties ./
[[email protected] config]$
[[email protected] config]$ # With the folder empty exept for log4j.properties and dfc.properties files, dmqdocbroker is working
[[email protected] config]$ date; time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping; date
Wed Apr 14 07:47:17 UTC 2021
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0200.0080
Using specified port: 1489
Successful reply from docbroker at host (cs-0) on port(1490) running software version (16.4.0200.0256 Linux64).

real    0m4.280s
user    0m8.161s
sys     0m0.729s
Wed Apr 14 07:47:21 UTC 2021
[[email protected] config]$
[[email protected] config]$ ls -l
total 20
drwxr-x--- 8 dmadmin dmadmin 4096 Apr 14 07:47 Shared
-rw-r----- 1 dmadmin dmadmin 2153 Apr 14 07:47 dfc.keystore
-rw-rw-r-- 1 dmadmin dmadmin  481 Jul 26  2020 dfc.properties
-rwxr-x--- 1 dmadmin dmadmin 1242 Jul 26  2020 log4j.properties
drwxr-x--- 6 dmadmin dmadmin 4096 Apr 14 07:46 test
[[email protected] config]$
[[email protected] config]$ rm -rf Shared/ dfc.keystore
[[email protected] config]$
[[email protected] config]$ mv test/dfc.keystore ./
[[email protected] config]$ mv test/dfcfull.properties ./
[[email protected] config]$ mv test/dfc.properties* ./
[[email protected] config]$ mv test/log4j.properties* ./
[[email protected] config]$ mv test/ServerApps ./
[[email protected] config]$ mv test/Shared ./
[[email protected] config]$ mv test/acs ./
[[email protected] config]$ mv test/bpm ./
[[email protected] config]$
[[email protected] config]$ ls -l
total 140
drwxr-x--- 7 dmadmin dmadmin  4096 Jul 26  2020 ServerApps
drwxr-x--- 9 dmadmin dmadmin  4096 Jul 26  2020 Shared
drwxr-x--- 7 dmadmin dmadmin  4096 Jul 26  2020 acs
drwxr-x--- 7 dmadmin dmadmin  4096 Jul 26  2020 bpm
-rw-rw-r-- 1 dmadmin dmadmin  2152 Jul 26  2020 dfc.keystore
-rw-rw-r-- 1 dmadmin dmadmin   481 Jul 26  2020 dfc.properties
-rw-rw---- 1 dmadmin dmadmin    70 Jul 22  2020 dfc.properties.bak.0
-rwxr-x--- 1 dmadmin dmadmin   242 Jul 26  2020 dfc.properties.bak.1
-rw-rw-r-- 1 dmadmin dmadmin   271 Jul 26  2020 dfc.properties.bak.2
-rw-rw-r-- 1 dmadmin dmadmin   323 Jul 26  2020 dfc.properties.bak.3
-rw-rw-r-- 1 dmadmin dmadmin   481 Jul 26  2020 dfc.properties.bak.4
-rw-rw-r-- 1 dmadmin dmadmin   482 Jul 26  2020 dfc.properties.bak.5
-rwxrwx--- 1 dmadmin dmadmin 79268 Jul 22  2020 dfcfull.properties
-rwxr-x--- 1 dmadmin dmadmin  1242 Jul 26  2020 log4j.properties
drwxr-x--- 2 dmadmin dmadmin  4096 Apr 14 07:51 test
[[email protected] config]$
[[email protected] config]$ ls -l test/
total 0
-rwxr-x--- 1 dmadmin dmadmin 0 Jul 22  2020 dbor.properties
-rw-rw-r-- 1 dmadmin dmadmin 0 Jul 26  2020 dbor.properties.lck
[[email protected] config]$
[[email protected] config]$
[[email protected] config]$ # With the full folder except the dbor files, dmqdocbroker is still working
[[email protected] config]$ date; time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping; date
Wed Apr 14 07:51:30 UTC 2021
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0200.0080
Using specified port: 1489
Successful reply from docbroker at host (cs-0) on port(1490) running software version (16.4.0200.0256 Linux64).

real    0m3.501s
user    0m6.632s
sys     0m0.666s
Wed Apr 14 07:51:34 UTC 2021
[[email protected] config]$
[[email protected] config]$ mv test/dbor.properties* ./
[[email protected] config]$
[[email protected] config]$ # With the dbor files back, dmqdocbroker isn't working anymore
[[email protected] config]$ date; time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping; date
Wed Apr 14 07:51:56 UTC 2021
^C
real    0m30.682s
user    0m5.001s
sys     0m0.424s
Wed Apr 14 07:52:27 UTC 2021
[[email protected] config]$
[[email protected] config]$ mv dbor.properties.lck test/
[[email protected] config]$
[[email protected] config]$ # Removing just the dbor files again, dmqdocbroker is working again
[[email protected] config]$ date; time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping; date
Wed Apr 14 07:52:36 UTC 2021
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0200.0080
Using specified port: 1489
Successful reply from docbroker at host (cs-0) on port(1490) running software version (16.4.0200.0256 Linux64).

real    0m3.185s
user    0m5.546s
sys     0m0.578s
Wed Apr 14 07:52:39 UTC 2021
[[email protected] config]$
[[email protected] config]$ ll
total 140
drwxr-x--- 7 dmadmin dmadmin  4096 Jul 26  2020 ServerApps
drwxr-x--- 9 dmadmin dmadmin  4096 Jul 26  2020 Shared
drwxr-x--- 7 dmadmin dmadmin  4096 Jul 26  2020 acs
drwxr-x--- 7 dmadmin dmadmin  4096 Jul 26  2020 bpm
-rwxr-x--- 1 dmadmin dmadmin     0 Jul 22  2020 dbor.properties
-rw-r----- 1 dmadmin dmadmin     0 Apr 14 07:52 dbor.properties.lck
-rw-rw-r-- 1 dmadmin dmadmin  2152 Jul 26  2020 dfc.keystore
-rw-rw-r-- 1 dmadmin dmadmin   481 Jul 26  2020 dfc.properties
-rw-rw---- 1 dmadmin dmadmin    70 Jul 22  2020 dfc.properties.bak.0
-rwxr-x--- 1 dmadmin dmadmin   242 Jul 26  2020 dfc.properties.bak.1
-rw-rw-r-- 1 dmadmin dmadmin   271 Jul 26  2020 dfc.properties.bak.2
-rw-rw-r-- 1 dmadmin dmadmin   323 Jul 26  2020 dfc.properties.bak.3
-rw-rw-r-- 1 dmadmin dmadmin   481 Jul 26  2020 dfc.properties.bak.4
-rw-rw-r-- 1 dmadmin dmadmin   482 Jul 26  2020 dfc.properties.bak.5
-rwxrwx--- 1 dmadmin dmadmin 79268 Jul 22  2020 dfcfull.properties
-rwxr-x--- 1 dmadmin dmadmin  1242 Jul 26  2020 log4j.properties
drwxr-x--- 2 dmadmin dmadmin  4096 Apr 14 07:52 test
[[email protected] config]$
[[email protected] config]$ diff dbor.properties.lck test/dbor.properties.lck
[[email protected] config]$

 

So as you can see above (it’s rather long but I wanted to put all the evidences I gathered because I still cannot believe this is the cause of the issue), just removing/renaming the empty file “dbor.properties.lck” which was there, untouched, since almost 9 months is sufficient to have the dmqdocbroker/iapi/idql working again… Trying to put back the old empty file, the issue will come back. It’s the “same” file, same content (empty), same file format, everything… The only difference would be the Inode of course and the creation/modification dates.

After some more investigations, the issue appeared to be on the NAS behind which was still having a lock on the file, somehow. For information, I also had the same behavior on a second environment but with the file “$DOCUMENTUM/config/ServerApps/identityInterprocessMutex.lock” this time… So if that even happen to you, take a look at these lock files under $DOCUMENTUM/config and make sure there are no problems with the storage.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Morgan Patou
Morgan Patou

Technology Leader ECM & Senior Consultant