Documentum Upgrade – Castor is locked!

One of the best practices while doing upgrade of Documentum is to unlock the various repository objects before starting. This will make sure that if the upgrade process requires an update of some objects, then it will succeed and not cause you issues because you are missing some pieces. This also brings a second benefit: it means that if you ever see locked objects after the upgrade, then something probably went wrong somewhere.

In a recent upgrade, I just had this case so it was the perfect example to demonstrate why this can be useful. Before the upgrade, there were no locked objects. You might have some user’s documents, which is obviously fine and you shouldn’t touch them normally, unless you have some external reasons:

[dmadmin@cs-0 ~]$ repo=REPO1
[dmadmin@cs-0 ~]$
[dmadmin@cs-0 ~]$ iapi ${repo} -U${USER} -Pxxx << EOF
> ?,c,select r_object_id, object_name, r_object_type, r_creation_date, r_modify_date, r_lock_owner, r_lock_machine, r_lock_date from dm_sysobject where r_lock_owner!=' ';
> EOF


        EMC Documentum iapi - Interactive API interface
        (c) Copyright EMC Corp., 1992 - 2015
        All rights reserved.
        Client Library Release 7.2.0150.0154


Connecting to Server using docbase REPO1
[DM_SESSION_I_SESSION_START]info:  "Session 010f1234809825ae started for user dmadmin."


Connected to Documentum Server running Release 7.2.0160.0297  Linux64.Oracle
Session id is s0
API> r_object_id       object_name     r_object_type   r_creation_date     r_modify_date       r_lock_owner       r_lock_machine       r_lock_date
----------------  --------------  --------------  ------------------  ------------------  -----------------  -------------------  -----------------
(0 rows affected)

API> Bye
[dmadmin@cs-0 ~]$

Then after the upgrade, there were one locked item:

[dmadmin@cs-0 ~]$ iapi ${repo} -U${USER} -Pxxx << EOF
> ?,c,select r_object_id, object_name, r_object_type, r_creation_date, r_modify_date, r_lock_owner, r_lock_machine, r_lock_date from dm_sysobject where r_lock_owner!=' ';
> EOF


        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0200.0080


Connecting to Server using docbase REPO1
[DM_SESSION_I_SESSION_START]info:  "Session 010f12348099e600 started for user dmadmin."


Connected to OpenText Documentum Server running Release 16.4.0200.0256  Linux64.Oracle
Session id is s0
API> r_object_id       object_name     r_object_type   r_creation_date     r_modify_date       r_lock_owner       r_lock_machine       r_lock_date
----------------  --------------  --------------  ------------------  ------------------  -----------------  -------------------  -----------------
090f123480000344  Castor          dmc_jar         12/1/2018 08:56:05  12/1/2018 08:56:05  dmadmin            cs-0                 5/7/2020 11:12:12
(1 rows affected)

[dmadmin@cs-0 ~]$

As you can see above, it was an upgrade from 7.2 (the source version doesn’t matter for this issue) to 16.4 P20. The ‘Castor’ object has been locked during the upgrade but it has apparently not been unlocked and it hasn’t been updated either. Therefore, that’s clearly two indicators that something went wrong. Castor is a third party software included by OpenText to provide BPM runtime BOF aspects and TBOs. It is configured as part of the dm_bpmmodules_install EBS script. So, looking at the log file of the EBS script generated during the upgrade showed the following:

[dmadmin@cs-0 ~]$ cat $DOCUMENTUM/dba/config/${repo}/dm_bpmmodules_install.out
Error checking out of object 090f123480000344: [DM_API_E_NO_MATCH]error:  "There was no match in the docbase for the qualification: dmc_jar where object_name = 'BPM XML Schema'"


Failed to create dmc_jar object (Castor)
[dmadmin@cs-0 ~]$

Based on the generated log file, it is clear that something went wrong there… Looking into the repository object, the ‘BPM XML Schema‘ object has been created properly but somehow the error shows that it cannot find it?

[dmadmin@cs-0 ~]$ iapi ${repo} -U${USER} -Pxxx << EOF
> ?,c,select r_object_id, object_name, r_object_type, r_creation_date, r_modify_date, r_lock_owner, r_lock_machine, r_lock_date from dm_sysobject where object_name='BPM XML Schema' OR object_name='Castor';
> EOF


        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0200.0080


Connecting to Server using docbase REPO1
[DM_SESSION_I_SESSION_START]info:  "Session 010f12348099e62e started for user dmadmin."


Connected to OpenText Documentum Server running Release 16.4.0200.0256  Linux64.Oracle
Session id is s0
API> r_object_id       object_name     r_object_type   r_creation_date     r_modify_date       r_lock_owner       r_lock_machine       r_lock_date
----------------  --------------  --------------  ------------------  ------------------  -----------------  -------------------  -----------------
090f1234809915ae  BPM XML Schema  dmc_jar         5/7/2020 11:12:12   5/7/2020 11:12:12                                           nulldate
090f123480000344  Castor          dmc_jar         12/1/2018 08:56:05  12/1/2018 08:56:05  dmadmin            cs-0                 5/7/2020 11:12:12
(2 rows affected)

API> Bye
[dmadmin@cs-0 ~]$

Since this seemed strange, I opened the OpenText SR#4507002 for them to look into it because the ‘BPM XML Schema‘ object was created but somehow it isn’t found right after while updating the ‘Castor‘ one… If you look at the dates, it clearly shows that there is something fishy here ;). According to OpenText, this is because the message put on the log is actually not the correct error message but a previous message from the execution of the EBS script, so it hides the real issue. The dmbasic code is using an message queue for the errors and as part of the CS 16.4 P10, OpenText updated this EBS script to add an additional jar file apparently. As part of this, the initial message regarding the ‘DM_API_E_NO_MATCH‘ (which seems to be expected) is never flushed and therefore when there is a subsequent error related to the checkout, it is the first error that is displayed in the logs, which isn’t correct since it shows the no match on ‘BPM XML Schema‘ while actually trying to create the ‘Castor‘ object.

They are still working on a fix for this issue, which will probably be integrated into a next patch (Update: I got the feedback that it will be integrated in the CS 16.4 P29, planned for release at the end of September). Until then, you can fix the issue by simply executing the EBS script manually, just don’t forget to unlock the ‘Castor‘ object before executing the EBS script with something like “unlock,c,<r_object_id>” (use the r_object_id of the ‘Castor‘ object obviously). The execution of the EBS script is pretty straightforward:

[dmadmin@cs-0 ~]$ cd $DM_HOME/install/admin
[dmadmin@cs-0 admin]$
[dmadmin@cs-0 admin]$ dmbasic -f dm_bpmmodules_install.ebs -e Install -- ${repo} dmadmin xxx $DM_HOME/lib
[dmadmin@cs-0 admin]$
[dmadmin@cs-0 admin]$ iapi ${repo} -U${USER} -Pxxx << EOF
> ?,c,select r_object_id, object_name, r_object_type, r_creation_date, r_modify_date, r_lock_owner, r_lock_machine, r_lock_date from dm_sysobject where object_name='BPM XML Schema' OR object_name='Castor';
> EOF


        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0200.0080


Connecting to Server using docbase REPO1
[DM_SESSION_I_SESSION_START]info:  "Session 010f1234809ec2d5 started for user dmadmin."


Connected to OpenText Documentum Server running Release 16.4.0200.0256  Linux64.Oracle
Session id is s0
API> r_object_id       object_name     r_object_type   r_creation_date     r_modify_date       r_lock_owner       r_lock_machine       r_lock_date
----------------  --------------  --------------  ------------------  ------------------  -----------------  -------------------  -----------------
090f1234809ea485  BPM XML Schema  dmc_jar         5/12/2020 20:27:56  5/12/2020 20:27:56                                          nulldate
090f1234809ea48d  Castor          dmc_jar         5/12/2020 20:27:56  5/12/2020 20:27:56                                          nulldate
(2 rows affected)

API> Bye
[dmadmin@cs-0 admin]$

This time, the ‘Castor‘ object has been recreated properly with a new ID, at the exact same time as the ‘BPM XML Schema‘ one and it is not locked anymore. This looks much better, isn’t it? 😉