Infrastructure at your Service

Morgan Patou

Documentum Upgrade – dm_DataDictionaryPublisher with wrong target server

As part of the same migration & upgrade project I talked about in previous blogs already (corrupt lockbox & duplicate objects), I have seen a strange but consistent behavior in each upgrade from Documentum 7.x to 16.x versions. During this process, just before the upgrade, a cloning is done to a staging environment, to keep the source untouched.

 

As part of this cloning, an update of the target_server value of the dm_job objects is done to use the new host, obviously. Therefore, once the cloning is done, I’m sure that all the jobs, without exception, were using the correct target_server, according to our best practices. This means most of the jobs targeting ANY running server and then distributed jobs with some specific configurations. As you probably already understood via the title of this blog, one of the issue I faced in every single upgrade from 7.x to 16.x is that the Documentum upgrade installer is changing the target_server of some jobs and for one of them, it is actually causing an issue.

 

Checking the target_server after the upgrade showed that this parameter has been updated for several jobs, which therefore lost our best practices. However, the real problem was for the dm_DataDictionaryPublisher job:

[[email protected]_cs ~]$ repo="REPO1"
[[email protected]_cs ~]$ iapi ${repo} -Udmadmin -Pxxx << EOC
?,c,SELECT r_object_id, object_name, r_creation_date, r_modify_date, target_server FROM dm_job WHERE object_name='dm_DataDictionaryPublisher';
exit
EOC

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0200.0080

Connecting to Server using docbase REPO1
[DM_SESSION_I_SESSION_START]info:  "Session 010f123450373995 started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0200.0256  Linux64.Oracle
Session id is s0
API> r_object_id       object_name                 r_creation_date     r_modify_date       target_server
----------------  --------------------------  ------------------  ------------------  ----------------------------------------------------------------------------------------
080f123450000381  dm_DataDictionaryPublisher  11/8/2018 07:04:37  2/10/2020 17:13:54  [email protected]_cs
(1 row affected)

API> Bye
[[email protected]_cs ~]$

 

Assuming a repository name of REPO1, you can connect to it using just “${repo_name}“. However, you can also connect to a specific Content Server using “${repo_name}.${dm_server_config_name}“. For a Primary CS, it is therefore usually something like “REPO1.REPO1“. I assume that what is happening is that somewhere in the code, instead of using “REPO1” as repository name and appending “.REPO1” to it so that the job run on the Primary CS, they actually used a repository name which is already the Primary CS target and therefore you end-up with “[email protected]” in the target_server, which is wrong.

 

During the upgrade process, there are many things happening, as you can expect… One of these is the execution of the toolset.ebs script. This script is configuring the target_server on several OOTB Documentum jobs. It will create (or rather overwrite in case of an upgrade) the log file “$DOCUMENTUM/dba/config/REPO1/sysadmin.trc“. Looking into this file about changes done to the target_server parameters, you can see the following:

[[email protected]_cs ~]$ cd $DOCUMENTUM/dba/config/${repo}/
[[email protected]_cs REPO1]$ 
[[email protected]_cs REPO1]$ grep -E "dm_DataDictionaryPublisher|target_server" sysadmin.trc | sed 's,.*\.[sg]et(,(,'
("set,c,080f123450000359,target_server","[email protected]_cs") ==> true
("set,c,080f12345000035a,target_server","[email protected]_cs") ==> true
("set,c,080f12345000035b,target_server","[email protected]_cs") ==> true
("set,c,080f12345000035c,target_server","[email protected]_cs") ==> true
("set,c,080f12345000035d,target_server","[email protected]_cs") ==> true
("set,c,080f12345000035e,target_server","[email protected]_cs") ==> true
("set,c,080f12345000035f,target_server","[email protected]_cs") ==> true
("set,c,080f123450000360,target_server","[email protected]_cs") ==> true
("set,c,080f123450000361,target_server","[email protected]_cs") ==> true
("set,c,080f123450000362,target_server","[email protected]_cs") ==> true
("set,c,080f12345000036d,target_server","[email protected]_cs") ==> true
("set,c,080f12345000036e,target_server","[email protected]_cs") ==> true
("set,c,080f12345000036f,target_server","[email protected]_cs") ==> true
("set,c,080f123450000370,target_server","[email protected]_cs") ==> true
("set,c,080f123450000371,target_server","[email protected]_cs") ==> true
("set,c,080f123450000372,target_server","[email protected]_cs") ==> true
("set,c,080f123450000373,target_server","[email protected]_cs") ==> true
("set,c,080f123450000374,target_server","[email protected]_cs") ==> true
("set,c,080f123450000375,target_server","[email protected]_cs") ==> true
("set,c,080f123450000376,target_server","[email protected]_cs") ==> true
("retrieve,c,dm_job where object_name = 'dm_DataDictionaryPublisher'") ==> "080f123450000381"
("set,c,080f123450000381,target_server","dummy") ==> true
("retrieve,c,dm_method where object_name = 'dm_DataDictionaryPublisher'") ==> "100f123450000380"
("set,c,080f123450000382,target_server","[email protected]_cs") ==> true
("set,c,080f123450000383,target_server","[email protected]_cs") ==> true
("set,c,080f123450000384,target_server","[email protected]_cs") ==> true
[[email protected]_cs REPO1]$

 

During the upgrade process, the target_server is set to “dummy” by this script for the dm_DataDictionaryPublisher job. As you know, that’s definitively not a correct value for a target_server but this is expected in this case, most probably to prevent it from running (we deactivate all the jobs before migration anyway)… Later, as part of the step (from the logs) “[main] com.documentum.install.server.installanywhere.actions.cfs.DiWAServerProcessingScripts2 – Executing the Java Method Server Configuration Setup script“, the script named “dm_defaultevent_set.ebs” is executed and create the log file “$DOCUMENTUM/dba/config/REPO1/dm_defaultevent_set.out“. Looking inside this script, you can see the following:

[[email protected]_cs REPO1]$ grep -C5 "retrieve.*dm_DataDictionary" $DM_HOME/install/admin/dm_defaultevent_set.ebs
  TargetServer = docbase & "." & _
  dmAPIGet("get,s0,serverconfig,object_name") & _
  "@" & dmAPIGet("get,s0,serverconfig,r_host_name")


  ID = dmAPIGet("retrieve," & sess & ",dm_job where object_name = 'dm_DataDictionaryPublisher'")
  If ID <> "" Then
    buff = "set," & sess & "," & ID & ",target_server"
    ret% = dmAPISet(buff, TargetServer)
    buff = "set," & sess & "," & ID & ",run_now"
    ret% = dmAPISet(buff, "T")
[[email protected]_cs REPO1]$

 

Based on the above, it looks like the script is correct: it does set the target_server to be “${repo_name}.${dm_server_config_name}@${dm_server_config_hostname}“. Therefore, as I suspected & mentioned already above, it must be a problem caused by the fact that the “${repo_name}” used to execute this script isn’t “REPO1” but is instead “REPO1.REPO1“. To confirm that, simply compare the definition of the “docbase” variable from the script with the generated log file:

[[email protected]_cs REPO1]$ grep "Start running" $DM_HOME/install/admin/dm_defaultevent_set.ebs
    Print "Start running dm_defaultevent_set.ebs script on docbase " & docbase
[[email protected]_cs REPO1]$
[[email protected]_cs REPO1]$ cat dm_defaultevent_set.out
Connect to docbase REPO1.REPO1
Start running dm_defaultevent_set.ebs script on docbase REPO1.REPO1
Audited the dm_docbase_config for event dm_default_set
Setting the Data Dictionary job state to active
Successfully updated job object for DataDictionaryPublisher
Finished running dm_defaultevent_set.ebs...
Disconnect from the docbase.
[[email protected]_cs REPO1]$

 

The above confirms that the ebs script is indeed executed with a repository name of “REPO1.REPO1” which cause the wrong target_server in the end. Looking at the other log files created by the upgrade shows that there are apparently some discrepancies. The following log files seems to be OK: dm_crypto_upgrade_52.out, headstart.out, replicate_bootstrap.out, sysadmin.trc, aso… While these ones apparently might not (would needs some debugging to validate): dm_gwm_install.out?, dm_java_methods.out?, dm_defaultevent_set.out, aso…

 

I guess I will open a Service Reqest with the OpenText Support to get this looked at. It might not be a big issue if it’s just about the target_server of the dm_DataDictionaryPublisher job but I’m afraid there would be something else hidden in another script or object that I haven’t seen yet… The previous blog I wrote about duplicate JMS/ACS object (link at the top of this blog) might also be a consequence? Anyway, while waiting for a feedback from OpenText, we can just fix the issue for this job by re-applying the correct configuration (running on ANY):

[[email protected]_cs REPO1]$ iapi ${repo} -Udmadmin -Pxxx << EOC
?,c,UPDATE dm_job OBJECTS SET target_server=' ' WHERE object_name='dm_DataDictionaryPublisher';
?,c,SELECT r_object_id, object_name, r_creation_date, r_modify_date, target_server FROM dm_job WHERE object_name='dm_DataDictionaryPublisher';
exit
EOC

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0200.0080

Connecting to Server using docbase REPO1
[DM_SESSION_I_SESSION_START]info:  "Session 010f123450373a42 started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0200.0256  Linux64.Oracle
Session id is s0
API> objects_updated
---------------
              1
(1 row affected)
[DM_QUERY_I_NUM_UPDATE]info:  "1 objects were affected by your UPDATE statement."

API> ?,c,SELECT r_object_id, object_name, r_creation_date, r_modify_date, target_server FROM dm_job WHERE object_name='dm_DataDictionaryPublisher';
r_object_id       object_name                 r_creation_date     r_modify_date       target_server
----------------  --------------------------  ------------------  ------------------  --------------
080f123450000381  dm_DataDictionaryPublisher  11/8/2018 07:04:37  2/11/2020 07:04:45  
(1 row affected)

API> Bye
[[email protected]_cs REPO1]$

 

If I receive some important information from the OpenText Service Request, I will try to update this blog to add the details (like other potential issues linked).

 

Edit: I opened the SR#4485076 and OpenText confirmed the bug. They apparently already knew about the issue for dm_DataDictionaryPublisher since it has been fixed (normally) in 16.4 P23 released on 30-March-2020. They are still looking into potential issues with the other scripts.

 

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Morgan Patou
Morgan Patou

Technology Leader ECM & Senior Consultant