Documentum Upgrade – Corrupt Lockbox with CS 16.4

A couple months ago, I was working on an upgrade project from Documentum 7.x to 16.4. During this upgrade, I faced a few issues so I will try to write some blogs about all that in the coming weeks, starting today with an alleged “corrupt” Lockbox. This upgrade was part of a migration as well from Virtual Machines to Containers (using Kubernetes), therefore the upgrade was done on a staging environment build specifically for that using the source version. The cloning process was done properly and then the upgrade started.

Upgrading the binaries was done without error but there were some INFO messages that I found interesting:

[dmadmin@stg-cs ~]$ cd $DCTM_BINARY/logs/
[dmadmin@stg-cs logs]$ grep -i "lock" install.log
08:35:27,490  INFO [main] com.documentum.install.shared.actions.DiActionExtractSystemResourceTarget - Extracting java system resource upgradeLockbox.sh to $DM_HOME/bin/upgradeLockbox.sh
08:35:27,586  INFO [main] com.documentum.install.shared.actions.DiActionSetPermissionTarget - performing chmod 6750 $DM_HOME/bin/upgradeLockbox.sh
08:35:33,897  INFO [main] - Did not find existing lockbox file, don't need to upgrade existing lockbox!
[dmadmin@stg-cs logs]$

Since it was just an INFO message, I just kept a note about that to check it later and then I proceeded with the patching and repository upgrade. The patching went smoothly as well but it wasn’t the case for the repository upgrade which failed at the beginning:

[dmadmin@stg-cs logs]$ cd $DM_HOME/install/logs/
[dmadmin@stg-cs logs]$ cat install.log
11:07:01,522  INFO [main] com.documentum.install.shared.installanywhere.actions.InitializeSharedLibrary - The product name is: UniversalServerConfigurator
11:07:01,522  INFO [main] com.documentum.install.shared.installanywhere.actions.InitializeSharedLibrary - The product version is: 16.4.0000.0248
11:07:01,523  INFO [main] -
11:07:01,554  INFO [main] com.documentum.install.shared.installanywhere.actions.InitializeSharedLibrary - Done InitializeSharedLibrary ...
11:07:01,578  INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerInformation - Setting CONFIGURE_DOCBROKER value to TRUE for SERVER
11:07:01,578  INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerInformation - Setting CONFIGURE_DOCBASE value to TRUE for SERVER
11:07:02,581  INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerCheckEnvrionmentVariable - The installer was started using the dm_launch_server_config_program.sh script.
11:07:02,581  INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerCheckEnvrionmentVariable - The installer will determine the value of environment variable DOCUMENTUM.
11:07:05,581  INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerCheckEnvrionmentVariable - The installer will determine the value of environment variable PATH.
11:07:08,603  INFO [main] com.documentum.install.server.installanywhere.actions.DiWASilentConfigurationInstallationValidation - Start to validate docbase parameters.
11:07:08,610  INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerPatchExistingDocbaseAction - The installer will obtain all the DOCBASE on the machine.
11:07:10,618  INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerDocAppFolder - The installer will obtain all the DocApps which could be installed for the repository.
11:07:10,620  INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerLoadDocBaseComponentInfo - The installer will gather information about the component GR_REPO.
11:07:13,624  INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerCheckKeystoreStatusForOld - The installer will check old AEK key status.
11:07:15,173  INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerEnableLockBoxValidation - The installer will validate AEK/Lockbox fileds.
11:07:15,180  INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerCreateKeyStore - Upgrade docbase use keep aek unchanged in lockbox, will not re-create it
11:07:16,685 ERROR [main] com.documentum.install.server.installanywhere.actions.DiWAServerValidateLockboxPassphrase - Check AEK key passphrase failed
com.documentum.install.shared.common.error.DiException: Check AEK key passphrase failed
        at com.documentum.install.server.installanywhere.actions.DiWAServerValidateLockboxPassphrase.executeValidation(DiWAServerValidateLockboxPassphrase.java:133)
        at com.documentum.install.server.installanywhere.actions.DiWAServerValidateLockboxPassphrase.setup(DiWAServerValidateLockboxPassphrase.java:115)
        at com.documentum.install.shared.installanywhere.actions.InstallWizardAction.install(InstallWizardAction.java:73)
        at com.zerog.ia.installer.actions.CustomAction.installSelf(Unknown Source)
        ...
[dmadmin@stg-cs logs]$

At first, I thought that I provided the wrong passphrases for the AEK or Lockbox. Therefore, I tried to reset the server fingerprint to validate the Lockbox passphrase:

[dmadmin@stg-cs logs]$ dm_crypto_manage_lockbox -lockbox lockbox.lb -lockboxpassphrase ${lockbox_passphrase} -resetfingerprint
Lockbox lockbox.lb
Lockbox Path $DOCUMENTUM/dba/secure/lockbox.lb
The Lockbox is corrupt and failed to load.Lockbox open failed ▒pY
** Operation failed **
[dmadmin@stg-cs logs]$

After some investigation, I saw that I could actually work with the lockbox while using the CS 7.x binaries (initial version, already using the lockbox) but it wasn’t working while using the CS 16.4 binaries. So just setting the $DM_HOME to one or the other path (and reloading the environment) changed completely the behavior since one was working but not the other. Since it was therefore linked to the upgrade, I was thinking that maybe the “INFO” message above regarding the upgrade lockbox was actually an issue? So digging deeper, I saw two new parameters in the 16.4 upgrade silent properties:

SERVER.LOCKBOX_FILE_NAME1=lockbox.lb
SERVER.LOCKBOX_PASSPHRASE.PASSWORD1=xxx

These two parameters weren’t available as far as I know with the CS 7.3 and they only appeared with the CS 16.4. While trying to upgrade the binaries from 7.x to 16.4, you obviously don’t have the binaries already available in 16.4 and therefore you don’t have access to the silent properties files of 16.4 (unless you go dig them into the compressed archive) so I used the one from 7.x which was therefore without these two parameters. Is this the issue? Well not really… Obviously, if you specify these two parameters, then the lockbox will be upgraded with the binaries (I didn’t test it, hopefully it works…). However, it shouldn’t be mandatory, the repository upgrade should also perform the mandatory lockbox upgrade. The thing here is that we already have to provide the AEK & Lockbox passphrases in the repository upgrade so to avoid duplicating things unnecessarily, I wanted to do that on the repository upgrade directly, where it should be, from my point of view. Therefore, I assumed that this wasn’t really the issue here (I will come back on that later in this blog…).

Why is the repository upgrade failing then? Well as you can see, there is not enough information so far. I found that the passphrases that I used were indeed correct but it still failed because of the so called “corrupt” Lockbox. What I did first is to backup the lockbox and try to upgrade it myself:

[dmadmin@stg-cs logs]$ cd $DM_HOME/bin/
[dmadmin@stg-cs bin]$
[dmadmin@stg-cs bin]$ ./upgradeLockbox.sh lockbox.lb ${lockbox_passphrase}
Lockbox lockbox.lb
Lockbox Path $DOCUMENTUM/dba/secure/lockbox.lb
Renamed $DOCUMENTUM/dba/secure/lockbox.lb to $DOCUMENTUM/dba/secure/lockbox.lb.bak.2020-2-14.11.33.17
Renamed $DOCUMENTUM/dba/secure/lockbox.lb.FCD to $DOCUMENTUM/dba/secure/lockbox.lb.FCD.bak.2020-2-14.11.33.17
Creating initial Lockbox 4.0 file...
Reading old Lockbox file into buffer... $DOCUMENTUM/dba/secure/lockbox.lb.bak.2020-2-14.11.33.17
Setting pointer to Custom SSV providers...
Importing data from old Lockbox file into new Lockbox 4.0 handle...

Done!
[dmadmin@stg-cs bin]$ 
[dmadmin@stg-cs bin]$ diff $DOCUMENTUM/dba/secure/lockbox.lb $DOCUMENTUM/dba/secure/lockbox.lb.bak.2020-2-14.11.33.17
1c1
< 4.000000|...
 No newline at end of file
---
> 3.100000|...
 No newline at end of file
[dmadmin@stg-cs bin]$

As you can see above, the upgrade of the lockbox itself is working fine and it does change the version from 3.1 (used by CS 7.x) to 4.0 (used by CS 16.4). Running again the repository upgrade didn’t produce any error anymore so it was able to proceed and complete without any problem once the lockbox was manually upgraded. However, this is only a workaround so I wanted to get to the bottom of the issue and therefore with my second repository, I restored the initial lockbox, reloaded it in the memory with the CS 7.x binaries and enabled the DEBUG logs on the repository upgrade installer. Obviously, the DEBUG logs was quite huge so I won’t put everything in this blog but the relevant section is the following one:

[dmadmin@stg-cs bin]$ cd $DM_HOME/install/logs/
[dmadmin@stg-cs logs]$ cat install.log
12:02:28,957 DEBUG [main]  - ###################The variable is: LOG_IS_READY, value is: true
12:02:28,957 DEBUG [main]  - ###################The variable is: FORMATED_PRODUCT_VERSION_NUMBER, value is: 16.4.0000.0248
12:02:28,958  INFO [main] com.documentum.install.shared.installanywhere.actions.InitializeSharedLibrary - The product name is: UniversalServerConfigurator
12:02:28,958  INFO [main] com.documentum.install.shared.installanywhere.actions.InitializeSharedLibrary - The product version is: 16.4.0000.0248
12:02:28,958  INFO [main]  -
12:02:28,959 DEBUG [main]  - ###################The variable is: DCTM_INSTALLER_TEMP_DIR, value is: /tmp/731965.tmp
...
12:02:41,075 DEBUG [main] com.documentum.install.server.installanywhere.actions.DiWAServerCheckKeystoreStatusForOld - *******************Start action com.documentum.install.server.installanywhere.actions.DiWAServerCheckKeystoreStatusForOld***********************
12:02:41,075  INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerCheckKeystoreStatusForOld - The installer will check old AEK key status.
12:02:41,079 DEBUG [main]  - Before running the following command, the class path is /tmp/install.dir.13366/InstallerData:/tmp/install.dir.13366/InstallerData/installer.zip:$DM_HOME/dctm-server.jar:$DOCUMENTUM/dctm.jar:$DOCUMENTUM/config:$DM_HOME/bin:$DM_HOME/dctm-server.jar:$DOCUMENTUM/dctm.jar:$DOCUMENTUM/config:$DM_HOME/bin:
12:02:41,079 DEBUG [main]  - Before running the following command, the path is /usr/xpg4/bin:$JAVA_HOME/bin:$DM_HOME/bin:$DOCUMENTUM/dba:$ORACLE_HOME/bin:$JAVA_HOME/bin:$DM_HOME/bin:$DOCUMENTUM/dba:$ORACLE_HOME/bin:$DM_HOME/bin:$ORACLE_HOME/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/dmadmin/.local/bin:/home/dmadmin/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin
12:02:42,569 DEBUG [main]  - Command line is /bin/sh -c "$DM_HOME/bin/dm_crypto_create -check -noprompt -lockbox lockbox.lb -keyname CSaek >"$DOCUMENTUM/temp/installer/installlogs/tempKeyStoreOutput201016218bfd-9e6c-3441-a81e-697068322a7391246586e34ef995978.out" 2>&1" and start in /tmp/731965.tmp. The return code of this command is 3
12:02:42,569 DEBUG [main]  - After running the above command, the class path is /tmp/install.dir.13366/InstallerData:/tmp/install.dir.13366/InstallerData/installer.zip:$DM_HOME/dctm-server.jar:$DOCUMENTUM/dctm.jar:$DOCUMENTUM/config:$DM_HOME/bin:$DM_HOME/dctm-server.jar:$DOCUMENTUM/dctm.jar:$DOCUMENTUM/config:$DM_HOME/bin:
12:02:42,569 DEBUG [main]  - After running the above command, the path is /usr/xpg4/bin:$JAVA_HOME/bin:$DM_HOME/bin:$DOCUMENTUM/dba:$ORACLE_HOME/bin:$JAVA_HOME/bin:$DM_HOME/bin:$DOCUMENTUM/dba:$ORACLE_HOME/bin:$DM_HOME/bin:$ORACLE_HOME/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/dmadmin/.local/bin:/home/dmadmin/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin
12:02:42,629 DEBUG [main]  - ###################The variable is: SERVER.OLD_KEYSTORE_STATUS, value is: NOT_EXIST
12:02:42,629 DEBUG [main]  - ###################The variable is: SERVER.KEYSTORE_STATUS, value is: NOT_EXIST
12:02:42,629 DEBUG [main] com.documentum.install.server.installanywhere.actions.DiWAServerCheckKeystoreStatusForOld - *******************************end of action********************************
12:02:42,633 DEBUG [main] com.documentum.install.server.installanywhere.actions.DiWAServerLoadValidAEKs - Start to resolve variable
12:02:42,633 DEBUG [main] com.documentum.install.server.installanywhere.actions.DiWAServerLoadValidAEKs - Start to check condition
12:02:42,633 DEBUG [main] com.documentum.install.server.installanywhere.actions.DiWAServerLoadValidAEKs - Start to setup
12:02:42,633 DEBUG [main] com.documentum.install.server.installanywhere.actions.DiWAServerLoadValidAEKs - *******************Start action com.documentum.install.server.installanywhere.actions.DiWAServerLoadValidAEKs***********************
12:02:42,633 DEBUG [main]  - ###################The variable is: COMMON.EXIST_AEK_FILES, value is: lockbox.lb
12:02:42,633 DEBUG [main] com.documentum.install.server.installanywhere.actions.DiWAServerLoadValidAEKs - *******************************end of action********************************
12:02:42,634 DEBUG [main] com.documentum.install.server.installanywhere.actions.DiWAServerEnableLockBoxValidation - Start to resolve variable
12:02:42,634 DEBUG [main] com.documentum.install.server.installanywhere.actions.DiWAServerEnableLockBoxValidation - Start to check condition
12:02:42,634 DEBUG [main] com.documentum.install.server.installanywhere.actions.DiWAServerEnableLockBoxValidation - Start to setup
12:02:42,634 DEBUG [main] com.documentum.install.server.installanywhere.actions.DiWAServerEnableLockBoxValidation - *******************Start action com.documentum.install.server.installanywhere.actions.DiWAServerEnableLockBoxValidation***********************
12:02:42,634  INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerEnableLockBoxValidation - The installer will validate AEK/Lockbox fileds.
12:02:42,637 DEBUG [main] com.documentum.install.server.installanywhere.actions.DiWAServerEnableLockBoxValidation - *******************************end of action********************************
12:02:42,638 DEBUG [main] com.documentum.install.server.installanywhere.actions.DiWAServerCreateKeyStore - Start to resolve variable
12:02:42,639 DEBUG [main] com.documentum.install.server.installanywhere.actions.DiWAServerCreateKeyStore - Start to check condition
12:02:42,639 DEBUG [main] com.documentum.install.server.installanywhere.actions.DiWAServerCreateKeyStore - Start to setup
12:02:42,639 DEBUG [main] com.documentum.install.server.installanywhere.actions.DiWAServerCreateKeyStore - *******************Start action com.documentum.install.server.installanywhere.actions.DiWAServerCreateKeyStore***********************
12:02:42,639 DEBUG [main]  - ###################The variable is: SERVER.KEYSTORE_FILE, value is: $DOCUMENTUM/dba/secure/CSaek
12:02:42,642  INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerCreateKeyStore - Upgrade docbase use keep aek unchanged in lockbox, will not re-create it
12:02:42,642 DEBUG [main] com.documentum.install.server.installanywhere.actions.DiWAServerCreateKeyStore - *******************************end of action********************************
12:02:42,643 DEBUG [main] com.documentum.install.server.installanywhere.actions.DiWAServerValidateLockboxPassphrase - Start to resolve variable
12:02:42,644 DEBUG [main] com.documentum.install.server.installanywhere.actions.DiWAServerValidateLockboxPassphrase - Start to check condition
12:02:42,644 DEBUG [main] com.documentum.install.server.installanywhere.actions.DiWAServerValidateLockboxPassphrase - Start to setup
12:02:42,644 DEBUG [main] com.documentum.install.server.installanywhere.actions.DiWAServerValidateLockboxPassphrase - *******************Start action com.documentum.install.server.installanywhere.actions.DiWAServerValidateLockboxPassphrase***********************
12:02:42,644 DEBUG [main]  - Before running the following command, the class path is /tmp/install.dir.13366/InstallerData:/tmp/install.dir.13366/InstallerData/installer.zip:$DM_HOME/dctm-server.jar:$DOCUMENTUM/dctm.jar:$DOCUMENTUM/config:$DM_HOME/bin:$DM_HOME/dctm-server.jar:$DOCUMENTUM/dctm.jar:$DOCUMENTUM/config:$DM_HOME/bin:
12:02:42,644 DEBUG [main]  - Before running the following command, the path is /usr/xpg4/bin:$JAVA_HOME/bin:$DM_HOME/bin:$DOCUMENTUM/dba:$ORACLE_HOME/bin:$JAVA_HOME/bin:$DM_HOME/bin:$DOCUMENTUM/dba:$ORACLE_HOME/bin:$DM_HOME/bin:$ORACLE_HOME/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/dmadmin/.local/bin:/home/dmadmin/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin
12:02:44,112 DEBUG [main]  - Command line is /bin/sh -c "dm_crypto_create -check -noprompt -passphrase ****** -lockbox lockbox.lb -keyname CSaek > /tmp/731965.tmp/dm_crypto_create_check362671.log" and start in $DM_HOME/bin. The return code of this command is 3
12:02:44,113 DEBUG [main]  - After running the above command, the class path is /tmp/install.dir.13366/InstallerData:/tmp/install.dir.13366/InstallerData/installer.zip:$DM_HOME/dctm-server.jar:$DOCUMENTUM/dctm.jar:$DOCUMENTUM/config:$DM_HOME/bin:$DM_HOME/dctm-server.jar:$DOCUMENTUM/dctm.jar:$DOCUMENTUM/config:$DM_HOME/bin:
12:02:44,113 DEBUG [main]  - After running the above command, the path is /usr/xpg4/bin:$JAVA_HOME/bin:$DM_HOME/bin:$DOCUMENTUM/dba:$ORACLE_HOME/bin:$JAVA_HOME/bin:$DM_HOME/bin:$DOCUMENTUM/dba:$ORACLE_HOME/bin:$DM_HOME/bin:$ORACLE_HOME/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/dmadmin/.local/bin:/home/dmadmin/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin
12:02:44,165 DEBUG [main] com.documentum.install.shared.common.error.DiException - Check AEK key passphrase failed
12:02:44,166 ERROR [main] com.documentum.install.server.installanywhere.actions.DiWAServerValidateLockboxPassphrase - Check AEK key passphrase failed
com.documentum.install.shared.common.error.DiException: Check AEK key passphrase failed
        at com.documentum.install.server.installanywhere.actions.DiWAServerValidateLockboxPassphrase.executeValidation(DiWAServerValidateLockboxPassphrase.java:133)
        at com.documentum.install.server.installanywhere.actions.DiWAServerValidateLockboxPassphrase.setup(DiWAServerValidateLockboxPassphrase.java:115)
        at com.documentum.install.shared.installanywhere.actions.InstallWizardAction.install(InstallWizardAction.java:73)
        at com.zerog.ia.installer.actions.CustomAction.installSelf(Unknown Source)
        ...
[dmadmin@stg-cs logs]$

As you can see above, the first strange thing is that the installer has a “return code of this command is 3” while trying to check the “old AEK key status“. If you look at the command that is printed in the DEBUG logs, you can see this: $DM_HOME/bin/dm_crypto_create -check -noprompt -lockbox lockbox.lb -keyname CSaek. This is actually correct if you are using the default AEK passphrase but it’s not if you are using a custom passphrase… If you look a little bit lower in the DEBUG logs, you can see the correct command used later to check the AEK: dm_crypto_create -check -noprompt -passphrase ****** -lockbox lockbox.lb -keyname CSaek. Maybe it’s just a logging topic where the passphrase was completely removed from the first command while it was replaced by “******” on the second command? Anyway, I found that interesting to note…

Since I do not have access to the source code, I needed OpenText to confirm if this was potentially the issue or if it was the first thing I mentioned earlier related to the properties in the CS Upgrade silent properties file. I opened the OpenText Service Request #4436375 and after two months, I got the feedback that the lockbox upgrade can only be done during the CS binaries upgrade. This doesn’t make any sense to me. As I said previously, the lockbox is related to a repository, it has nothing to do with the binaries. In addition, you already need to provide lockbox details during the repository upgrade so why doing this in the binaries upgrade which therefore needs you to enter the passwords on yet another silent properties file? An enhancement request would have been possible but as you probably know, Documentum 16.7 removed the support for the Lockbox so it does only impact Documentum 16.4 and this will therefore not change anymore. I guess I will stick with my workaround in our custom silent script package so that the lockbox if upgraded, if needed (present in 7.x & target in 16.4), just before running the repository upgrade. This is all done automatically in our Ansible playbooks so that we can just execute one command and we have the upgrade done from A to Z, successfully.