Documentum story – Disable FIPS-140 on a CS 7.2 P05 in order to connect to a 6.7 SP2 repository

In this blog and for a customer’s requirement (Data Migration), I had to temporarily disable the FIPS on a Content Server 7.2 P05 in order to connect to a repository 6.7 SP2. In fact after adding the 6.7 repository to the docbroker projection list of the CS 7.2, the customer was not able to login and had the following error message:

[dmadmin@content_server_01 ~]$ idql repo67
Please enter a user (dmadmin): aberbier 
Please enter password for aberbier: xxxxx

EMC Documentum idql - Interactive document query interface
(c) Copyright EMC Corp., 1992 - 2015
All rights reserved.
Client Library Release 7.2.0050.0084

Connecting to Server using docbase repo67
Could not connect
[DM_SESSION_E_RPC_ERROR]error: "Server communication failure "

javax.net.ssl.SSLException: Client does not support server chosen protocol: SSLv3

com.rsa.sslj.x.g: Client does not support server chosen protocol: SSLv3

To understand what happened, we had to retrieve the docbase map of the docbroker using the dmqdocbroker utility:

[dmadmin@content_server_01 ~]$ dmqdocbroker -i
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 7.2.0050.0084
Targeting current host
Targeting port 1489
---- dmqdocbroker: (TARGET HOST: content_server_01) ----
p) Ping (test connectivity to) the docbroker
d) Get a docbase map
s) Get a server map
n) Get next largest docbase id
l) lookup a docbase id
o) find all open servers for a docbase
h) Set the host name for the docbroker
e) exit
Enter an option (i.e. letter)> d
**************************************************
** D O C B R O K E R I N F O **
**************************************************
Docbroker host : content_server_01
Docbroker port : 1490
Docbroker network address : INET_ADDR: 02 5c3 93a73d7f content_server_01 172.1.1.2
Docbroker version : 7.2.0050.0214 Linux64
**************************************************
** D O C B A S E I N F O **
**************************************************
--------------------------------------------
Docbase name : gr_dbi
Docbase id : 1103520
Docbase description : dbi services Development Global Repository
Govern docbase :
Federation name :
Server version : 7.2.0050.0214 Linux64.Oracle
Docbase Roles : Global Registry
Docbase Dormancy Status :
--------------------------------------------
Docbase name : repo67
Docbase id : 1003563
Docbase description :
Govern docbase : gov
Federation name : fed
Server version : 6.7.2020.0057 AIX.Oracle
Docbase Roles :
Docbase Dormancy Status :

We saw that the repository we were trying to connect was hosted on an AIX server which was using the Oracle RDBMS(????). With the help of the EMC support and as per already faced issues with the communication between the content server, the web application server and web application client, we observed that the issue was related to the FIPS 140 which didn’t allow the communication with weak protocol as SSLv3: only TLS and higher are supported with a Content Server 7.2.

To work around this issue, we decided to temporarily disable the FIPS 140 on the CS 7.2 during the time of the data migration. Please find below the procedure that we applied to disable and re-enable the FIPS.

Disable FIPS
==========

1. Login to the CS 7.2
2. Change the current user to the Installation Owner
3. Backup the current java.security file: cp $JAVA_HOME/jre/lib/security/java.security $JAVA_HOME/jre/lib/security/java.security.bck-FIPS-Enabled
4. Edit the java.security to disable FIPS: vi $JAVA_HOME/jre/lib/security/java.security
5. Add the following line at the end of the file : com.rsa.cryptoj.jsafe.fips140initialmode=NON_FIPS140_MODE
6. Save the file
7. Restart the Content Server

After doing that, we were able to properly login to the remote 6.7 SP2 repository:

[dmadmin@content_server_01 ~]$ idql repo67
Please enter a user (dmadmin): aberbier
Please enter password for aberbier: xxxxx

EMC Documentum idql - Interactive document query interface
(c) Copyright EMC Corp., 1992 - 2015
All rights reserved.
Client Library Release 7.2.0050.0084

Connecting to Server using docbase repo67
[DM_SESSION_I_SESSION_START]info: "Session 010000cf805192b0 started for user Arnaud Berbier. "

Connected to Documentum Server running Release 6.7.2020.0057 AIX.Oracle
1>

When the data migration has been properly completed and to stay in safe mode, we recommended to revert the changes back as soon as possible.

Revert back the changes
===================

1. Login to the CS 7.2
2. Change the current user to the Installation Owner
3. Revert the java.security backup file: mv $JAVA_HOME/jre/lib/security/java.security.bck-FIPS-Enabled $JAVA_HOME/jre/lib/security/java.security
5. Restart the Content Server

We strongly recommend not leaving the FIPS 140 disabled as we don’t really know the impact from a security point of view.

In this blog post, we provided a procedure to temporarily disable and re-activate the FIPS 140 on a Content Server 7.2 P05 in order to connect to a 6.7 SP2.