Recently, I’ve been facing an issue more and more frequently where a Repository cannot register itself to a local/remote Docbroker. The root cause of the issue is simply that the hostname used for the projection to the Docbroker is too long which prevents the Repository to properly register to it. I’ve seen it recently because when working with Kubernetes, you can pretty quickly end-up with very very long hostnames… The issue isn’t linked to K8s in itself, it just helps to reproduce the issue. Alternatively, you can do the same thing by just tweeking the /etc/hosts file to lenghten a name that can then be used for the projection.

Alright so let’s demonstrate this issue. The simplest way to reproduce the issue is to have a FQDN that is strictly longer than 59 characters and a short hostname that is shorter:

[dmadmin@documentum-server-0 ~]$ hostname | wc -m
20
[dmadmin@documentum-server-0 ~]$ hostname
documentum-server-0
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ hostname -f | wc -m
63
[dmadmin@documentum-server-0 ~]$ hostname -f
documentum-server-0.documentum-server.dbi-ns.svc.cluster.local
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ grep "^host=" $DOCUMENTUM/dba/dm_launch_Docbroker
host=documentum-server-0
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ dmqdocbroker -t `hostname` -p 1489 -c ping
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0110.0058
Using specified port: 1489
Successful reply from docbroker at host (documentum-server-0) on port(1490) running software version (16.4.0110.0167 Linux64).
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ dmqdocbroker -t `hostname -f` -p 1489 -c ping
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0110.0058
Using specified port: 1489
Successful reply from docbroker at host (documentum-server-0) on port(1490) running software version (16.4.0110.0167 Linux64).
[dmadmin@documentum-server-0 ~]$

 

As you can see above, everything seems fine. Let’s see what is the current configuration and if it works properly:

[dmadmin@documentum-server-0 ~]$ grep "^host=" $DOCUMENTUM/dba/dm_launch_Docbroker
host=documentum-server-0
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ $DOCUMENTUM/dba/dm_stop_Docbroker
./dmshutdown 16.4.0000.0248  Linux64 Copyright (c) 2018. OpenText Corporation.
Shutdown request was processed by Docbroker on host documentum-server-0 (INET_ADDR: 01 2d3 01010164 documentum-server-0 1.1.1.100)
Reply status indicates a success: OK
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ $DOCUMENTUM/dba/dm_launch_Docbroker
starting connection broker on current host: [documentum-server-0]
with connection broker log: [$DOCUMENTUM/dba/log/docbroker.documentum-server-0.1489.log]
connection broker pid: 19073
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ cat $DOCUMENTUM/dba/log/docbroker.documentum-server-0.1489.log
OpenText Documentum Connection Broker (version 16.4.0110.0167  Linux64)
Copyright (c) 2018. OpenText Corporation
2019-11-16T11:30:44.737512 [DM_DOCBROKER_I_START]info:  "Docbroker has started.  Process id: 19073"
2019-11-16T11:30:44.747745 [DM_DOCBROKER_I_REGISTERED_PORT]info:  "The Docbroker registered using port (1489)."
2019-11-16T11:30:44.747797 [DM_DOCBROKER_I_LISTENING]info:  "The Docbroker is listening on network address: (INET_ADDR: family: 2, port: 1489, host: documentum-server-0 (1.1.1.100, e01234ac))"
2019-11-16T11:30:44.747812 [DM_DOCBROKER_I_REGISTERED_PORT]info:  "The Docbroker registered using port (1490)."
2019-11-16T11:30:44.747828 [DM_DOCBROKER_I_LISTENING]info:  "The Docbroker is listening on network address: (INET_ADDR: family: 2, port: 1490, host: documentum-server-0 (1.1.1.100, e01234ac))"
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ grep -A3 "TARGET]" $DOCUMENTUM/dba/config/gr_repo/server.ini
[DOCBROKER_PROJECTION_TARGET]
host = documentum-server-0
port = 1489
proximity = 1
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ $DOCUMENTUM/dba/dm_start_gr_repo
starting Documentum server for repository: [gr_repo]
with server log: [$DOCUMENTUM/dba/log/gr_repo.log]
server pid: 19245
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ grep broker $DOCUMENTUM/dba/log/gr_repo.log
2019-11-16T11:31:48.749813      19245[19245]    0000000000000000        [DM_SERVER_I_START]info:  "Sending Initial Docbroker check-point "
2019-11-16T11:31:49.008865      19318[19318]    0112d68780000003        [DM_DOCBROKER_I_PROJECTING]info:  "Sending information to Docbroker located on host (documentum-server-0) with port (1490).  Information: (Config(gr_repo), Proximity(1), Status(Open), Dormancy Status(Active))."
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ dmqdocbroker -t `hostname` -p 1489 -c getdocbasemap
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0110.0058
Using specified port: 1489
**************************************************
**     D O C B R O K E R    I N F O             **
**************************************************
Docbroker host            : documentum-server-0
Docbroker port            : 1490
Docbroker network address : INET_ADDR: 01 2d3 01010164 documentum-server-0 1.1.1.100
Docbroker version         : 16.4.0110.0167  Linux64
**************************************************
**     D O C B A S E   I N F O                  **
**************************************************
--------------------------------------------
Docbase name        : gr_repo
Docbase id          : 1234567
Docbase description : dbi dev k8s gr
Govern docbase      :
Federation name     :
Server version      : 16.4.0110.0167  Linux64.Oracle
Docbase Roles       : Global Registry
Docbase Dormancy Status     :
--------------------------------------------
[dmadmin@documentum-server-0 ~]$

 

So with the above short name on both the Docbroker start script and the Repository projection (server.ini), you can see that it’s all working. Now let’s replicate the issue:

[dmadmin@documentum-server-0 ~]$ $DOCUMENTUM/dba/dm_shutdown_gr_repo
Stopping Documentum server for repository: [gr_repo]

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0110.0058

Connecting to Server using docbase gr_repo.gr_repo
[DM_SESSION_I_SESSION_START]info:  "Session 0112d68780012505 started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0110.0167  Linux64.Oracle
Session id is s0
API> shutdown,c,T,T
...
OK
API> exit
Bye
Waiting for 90 seconds for server pid, 19245, to disappear.

Sat Nov 16 11:35:35 UTC 2019: Waiting for shutdown of repository: [gr_repo]
Sat Nov 16 11:35:35 UTC 2019: checking for pid: 19245

Sat Nov 16 11:35:45 UTC 2019: Waiting for shutdown of repository: [gr_repo]
Sat Nov 16 11:35:45 UTC 2019: checking for pid: 19245

repository: [gr_repo] has been shutdown
checking that all children (19251 19253 19254 19275 19297 19319 19362 19387 19570) have shutdown
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ grep -A3 "TARGET]" $DOCUMENTUM/dba/config/gr_repo/server.ini
[DOCBROKER_PROJECTION_TARGET]
host = documentum-server-0
port = 1489
proximity = 1
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ sed -i "s,^host.*,host = `hostname -f`," $DOCUMENTUM/dba/config/gr_repo/server.ini
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ grep -A3 "TARGET]" $DOCUMENTUM/dba/config/gr_repo/server.ini
[DOCBROKER_PROJECTION_TARGET]
host = documentum-server-0.documentum-server.dbi-ns.svc.cluster.local
port = 1489
proximity = 1
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ grep broker $DOCUMENTUM/dba/log/gr_repo.log
2019-11-16T11:38:34.593108      19715[19715]    0000000000000000        [DM_SERVER_I_START]info:  "Sending Initial Docbroker check-point "
2019-11-16T11:38:34.655402      19787[19787]    0112d68780000003        [DM_DOCBROKER_I_PROJECTING]info:  "Sending information to Docbroker located on host (documentum-server-0.documentum-server.dbi-ns.svc.cluster.lo) with port (1489).  Information: (Config(gr_repo), Proximity(1), Status(Open), Dormancy Status(Active))."
[dmadmin@documentum-server-0 ~]$

 

The message seems correct, right? You might be fooled by the message but if you look carefully at it, you can actually see that the host in the parenthesis is cut. The value is “documentum-server-0.documentum-server.dbi-ns.svc.cluster.lo” instead of “documentum-server-0.documentum-server.dbi-ns.svc.cluster.local“. That’s 59 characters exactly and nothing more. Since our FQDN is longer, then the Repository cut the hostname and keep only the first 59 characters. If you look at the log file directly of if you include the error messages as well in the grep (with the -i option for example for the uppercase BROKER errors), then you can see the failure:

[dmadmin@documentum-server-0 ~]$ grep -i broker $DOCUMENTUM/dba/log/gr_repo.log
2019-11-16T11:38:34.593108      19715[19715]    0000000000000000        [DM_SERVER_I_START]info:  "Sending Initial Docbroker check-point "
2019-11-16T11:38:34.655141      19787[19787]    0112d68780000003        [DM_DOCBROKER_E_NETWORK_ERROR]error:  "An error occured performing a network operation: ((112) Error opening connection).  Network specific error: ( errno: 0, message: Success)."
2019-11-16T11:38:34.655364      19787[19787]    0112d68780000003        [DM_DOCBROKER_E_CONNECT_FAILED_EX]error:  "Unable to connect to DocBroker. Clients please check your dfc.properties file for a correct host.  Server please check your server.ini or target attributes in server config.  Network address: (INET_ADDR: family: 2, port: 1489, host: documentum-server-0.documentum-server.dbi-ns.svc.cluster.lo (0.0.0.0, 00000000))."
2019-11-16T11:38:34.655402      19787[19787]    0112d68780000003        [DM_DOCBROKER_I_PROJECTING]info:  "Sending information to Docbroker located on host (documentum-server-0.documentum-server.dbi-ns.svc.cluster.lo) with port (1489).  Information: (Config(gr_repo), Proximity(1), Status(Open), Dormancy Status(Active))."
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ dmqdocbroker -t `hostname` -p 1489 -c getdocbasemap
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0110.0058
Using specified port: 1489
**************************************************
**     D O C B R O K E R    I N F O             **
**************************************************
Docbroker host            :
Docbroker port            :
Docbroker network address :
Docbroker version         :
**************************************************
**     D O C B A S E   I N F O                  **
**************************************************
--------------------------------------------
[dmadmin@documentum-server-0 ~]$

 

So as you can see above, just putting a hostname too long in the server.ini (or inside the Repository) will prevent it to project itself to the Docbroker and therefore you will never be able to work or even stop the Repository (kill is the only option). Therefore, take care that the hostname you are using for the projection isn’t too long.

 

Post Views: 682
Thumbnail [60x60]
by
Morgan Patou