Infrastructure at your Service

Morgan Patou

Documentum – SSL Certificate based secure communication setup

Around four years ago, I did a few presentations, here, in Switzerland about “Security & Documentum”. In there, I talked about a lot of different subjects related to both security and Documentum (you guessed it…) like: ciphers, SHA, FIPS & JCE, Documentum & DFC connect mode (characteristics, ciphers, protocols, encryptions, limitations), Documentum & DFC encryptions in transit and at rest (AEK/DBK/LTK/FSK/FEK, TCS, CS Lockbox, D2 Lockbox vs D2 Keystore, Passwords encryption and decryption), and some other topics (HTTPS on WebLogic, JBoss/WildFly, Tomcat, best practices for security, LDAPS support, FIPS 140-2 support and compliance).

 

“Why the hell are you talking about presentations you gave 4 years ago?”. Good question, thank you! This presentation was really dense so all I could do was just put an example of the configuration files needed for real SSL Certificate based secure communication but not how exactly to reach this point. I talked about this configuration in several blogs already but never took the time to explain/show it from A to Z. So, that’s what I’m going to do here because without being able to create the SSL Certificate and trust stores, you will probably have some trouble to really configure Documentum to use the real-secure mode (in opposition to the default-secure, which is using anonymous and therefore not fully secure).

 

In this blog, I will use self-signed SSL Certificate only. It is possible to use CA signed SSL Certificate, the only thing it would change is that you would need to set the trust chain into the different trust stores instead of the self-signed SSL Certificate. This has pros and cons however… This means it is easier to automate because a CA trust chain is a public SSL Certificate and therefore in case you are in a CI/CD infrastructure, you can easily create the needed Documentum trust stores from anywhere (any pods, any containers, any VMs, aso…). However, that also means that anybody with access to this trust chain can potentially create the needed files used by a DFC Client to talk to your Docbroker and Repositories. That might or might not be a problem for you so I will let you decide on that. On the other hand, using a self-signed SSL Certificate makes it more difficult to gain access to the certificates (unless you are storing it in a public and open location of course) but at the same time, this complicates a little bit the setup for remote DFC Clients since you will need to share, somehow, the Docbroker and Repositories certificates in order to create a trust store for the DFC Clients.

 

I split the steps into different sections: one global definition of parameters & passwords and then one section per component. Please note that for the DFC Client section, I used the JMS. The same steps can be applied for any DFC Client, you will just need to have access to the needed input files. Please make sure that all components are shutdown when you start the configuration, to avoid expected errors: it will be easier to spot errors if something you expect to be working isn’t, if you don’t have hundreds of expected errors in the middle because all clients are still trying to use non-secure (or default-secure) modes. Alright, enough blabbering, let’s start with the setup.

 

I. Global setup/parameters

All the files needed for the Docbroker and Repositories setup needs to be put into the $DOCUMENTUM/dba/secure/ folder so all the commands will be executed in there directly. I defined here some environment variables that will be used by all the commands. The read commands will simply ask you to enter the needed password and press enter. Doing that will store the password into the environment variable (lb_pp, b_pwd, s_pwd and d_pwd). If you aren’t using any Lockbox (since deprecated since Documentum 16.7), just ignore the Lockbox part.

cd $DOCUMENTUM/dba/secure/
lb_name="lockbox.lb"
aek_name="CSaek"
b_name="docbroker"
s_name="contentserver"
d_name="dfc"

read -s -p "  ----> Please enter the ${lb_name} passphrase: " lb_pp

read -s -p "  ----> Please enter the ${b_name} related password: " b_pwd

read -s -p "  ----> Please enter the ${s_name} related password: " s_pwd

read -s -p "  ----> Please enter the ${d_name} related password: " d_pwd

echo "
Lockbox passphrase entered: ${lb_pp}
Broker password entered: ${b_pwd}
Server password entered: ${s_pwd}
DFC password entered: ${d_pwd}"

 

II. Docbroker setup – SSL Server only

In this section, we will create the certificate for the Docbroker, create the needed keystore (it needs to be PKCS12) and encrypt the keystore password. If you aren’t using any Lockbox, in the “dm_encrypt_password” command, just remove the two parameters related to it (and its associated value/password) and remove the “crypto_lockbox” from the Docbroker.ini file (or whatever the name of your file is).

openssl req -x509 -days 1096 -newkey rsa:2048 -keyout ${b_name}.key -out ${b_name}.crt -subj "/C=CH/ST=Jura/L=Delemont/O=dbi services/OU=IT/CN=${b_name}" -passout pass:"${b_pwd}"

openssl pkcs12 -export -out ${b_name}.p12 -inkey ${b_name}.key -in ${b_name}.crt -name ${b_name} -descert -passin pass:"${b_pwd}" -passout pass:"${b_pwd}"

dm_encrypt_password -lockbox "${lb_name}" -lockboxpassphrase "${lb_pp}" -keyname "${aek_name}" -encrypt "${b_pwd}" -file ${b_name}.pwd

cp $DOCUMENTUM/dba/Docbroker.ini $DOCUMENTUM/dba/Docbroker.ini.orig

echo "[DOCBROKER_CONFIGURATION]
secure_connect_mode=secure
crypto_keyname=${aek_name}
crypto_lockbox=${lb_name}
keystore_file=${b_name}.p12
keystore_pwd_file=${b_name}.pwd" > $DOCUMENTUM/dba/Docbroker.ini

 

At this point, you can start the Docbroker and it should start only on the secure port, without errors. If there are still clients up&running, you will probably face a lot of handshake failure errors… It is possible to define the list of ciphers to use in the Docbroker.ini file (cipherlist=xxx:yyy:zzz) but if you do so, please make sure that all the SSL Clients (Repository and DFC Clients alike) that will talk to it does support this cipher as well.

 

III. Repository setup – SSL Server and SSL Client

In this section, we will create the certificate for the Repositories (each repo can have its own if you prefer), create the needed keystore (it needs to be PKCS12), create the needed trust store (it needs to be PKCS7) and encrypt the keystore password. If you aren’t using any Lockbox, in the “dm_encrypt_password” command, just remove the two parameters related to it (and its associated value/password). In case you have several Lockbox and AEK Key, you might want to retrieve their names from the server.ini directly (inside the loop) and then use these to encrypt the password, for each Repository, independently.

openssl req -x509 -days 1096 -newkey rsa:2048 -keyout ${s_name}.key -out ${s_name}.crt -subj "/C=CH/ST=Jura/L=Jura/O=dbi services/OU=IT/CN=${s_name}" -passout pass:"${s_pwd}"

openssl pkcs12 -export -out ${s_name}.p12 -inkey ${s_name}.key -in ${s_name}.crt -name ${s_name} -descert -passin pass:"${s_pwd}" -passout pass:"${s_pwd}"

dm_encrypt_password -lockbox "${lb_name}" -lockboxpassphrase "${lb_pp}" -keyname "${aek_name}" -encrypt "${s_pwd}" -file ${s_name}.pwd

openssl crl2pkcs7 -nocrl -certfile ${b_name}.crt -outform der -out ${s_name}-trust.p7b

for s_ini in $(ls $DOCUMENTUM/dba/config/*/server.ini); do
  cp ${s_ini} ${s_ini}.orig
  sed -i --follow-symlinks "/keystore_file/d" ${s_ini}
  sed -i --follow-symlinks "/keystore_pwd_file/d" ${s_ini}
  sed -i --follow-symlinks "/truststore_file/d" ${s_ini}
  sed -i --follow-symlinks "/cipherlist/d" ${s_ini}
  sed -i --follow-symlinks "/^crypto_keyname/a \truststore_file = ${s_name}-trust.p7b" ${s_ini}
  sed -i --follow-symlinks "/^crypto_keyname/a \keystore_pwd_file = ${s_name}.pwd" ${s_ini}
  sed -i --follow-symlinks "/^crypto_keyname/a \keystore_file = ${s_name}.p12" ${s_ini}
done

 

At this point, you can start the different Repositories and it should start and project itself to the Docbroker. However, The AgentExec should still fail to start properly because it should use the global dfc.properties of the Documentum Server, which wasn’t updated yet. So you might want to configure the global dfc.properties before starting the Repositories. It is possible to define the list of ciphers to use in the server.ini file (cipherlist=xxx:yyy:zzz) but if you do so, please make sure that all the SSL Clients (DFC Clients) that will talk to it and SSL Servers (Docbroker) it talks to does support this cipher as well.

 

IV. DFC Clients setup (JMS, IndexAgent, DA, D2, …) – SSL Client only

In this section, we will create the needed trust store (it needs to be JKS) and encrypt the trust store password. Regarding the password encryption, this command will work on any DFC Client, you will just need to add the dfc.jar in the classpath (for example on xPlore: -cp “$XPLORE_HOME/dfc/dfc.jar”) if you aren’t executing it on a Documentum Server.

openssl x509 -outform der -in ${b_name}.crt -out ${b_name}.der

openssl x509 -outform der -in ${s_name}.crt -out ${s_name}.der

$JAVA_HOME/bin/keytool -importcert -keystore ${d_name}-trust.jks -file ${b_name}.der -alias ${b_name} -noprompt -storepass ${d_pwd}

$JAVA_HOME/bin/keytool -importcert -keystore ${d_name}-trust.jks -file ${s_name}.der -alias ${s_name} -noprompt -storepass ${d_pwd}

d_pwd_enc=$($JAVA_HOME/bin/java com.documentum.fc.tools.RegistryPasswordUtils ${d_pwd})

cp $DOCUMENTUM/config/dfc.properties $DOCUMENTUM/config/dfc.properties.orig
sed -i '/dfc.session.secure_connect_default/d' $DOCUMENTUM/config/dfc.properties
sed -i '/dfc.security.ssl.use_existing_truststore/d' $DOCUMENTUM/config/dfc.properties
sed -i '/dfc.security.ssl.truststore/d' $DOCUMENTUM/config/dfc.properties
sed -i '/dfc.security.ssl.truststore_password/d' $DOCUMENTUM/config/dfc.properties

echo "dfc.session.secure_connect_default=secure
dfc.security.ssl.use_existing_truststore=false
dfc.security.ssl.truststore=$DOCUMENTUM/dba/secure/${d_name}-trust.jks
dfc.security.ssl.truststore_password=${d_pwd_enc}" >> $DOCUMENTUM/config/dfc.properties

 

This is technically the global dfc.properties of a Documentum Server and not really the JMS one but I assume almost everybody in the world is just including this one (using #include) for the dfc.properties of the JMS (ServerApps, acs, bpm, …), to avoid duplication of generic parameters/configurations at multiple locations and just manage them globally.

 

At this point, you can start the DFC Client and it should be able to communicate with the Docbroker and with the Repositories. As said before, if you already started the Repository, you might want to make sure that the AgentExec is running and if not, maybe restart the Repositories quickly.

 

Some final remarks on the SSL Certificate based secure configuration of Documentum:

  • Other Content Servers & Docbrokers (HA part) must re-use the exact same keystores (and therefore trust store as well in the end). Files must be sent to all other hosts and re-used exactly in the same way
  • Other DFC clients can use newly created files but in the end, it will contain the exact same content (either the self-signed Docbroker and Repositories certificates or the CA-signed trust chain)… Therefore, files can be sent to all DFC clients and re-used exactly in the same way as well
  • After the initial generation, you don’t need any of the key, crt or der files anymore so you can remove them for security reasons:
    • rm ${b_name}.key ${b_name}.crt ${b_name}.der ${s_name}.key ${s_name}.crt ${s_name}.der
  • I didn’t describe everything in full-length here, there are a bunch of other things and limitations to know before going into that direction so you will probably want to read the documentation carefully

 

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Morgan Patou
Morgan Patou

Technology Leader ECM & Senior Consultant