Documentum story – Thumbnail Server not starting if docbases are down

In this blog, I will talk about the Thumbnail Server. It’s a component of Documentum that you can install on the Content Server to generate… thumbnails! Basically what it does is that it will work in correlation with the ADTS/CTS in order to generate different kinds of previews of your document. For example jpeg_lres (low resolution) or jpeg_story (StoryBoard). You can define how many previews you want per document (1 preview per page, only the first page, aso…) and of which type. Then D2 can use these previews in a widget for the end-users to see what the document looks like. Apparently with D2 4.6, the Thumbnail Server and ADTS/CTS aren’t needed anymore to generate thumbnails but it was the case for D2 4.5 and previous versions.

That’s my first blog related to the Thumbnail Server because that’s actually a (the?) component of Documentum that is working pretty well without much issue so I absolutely wanted to explain the issue I faced and what has been done to solve that.

So let’s start with some background: I was working on a project where the TEST and PROD environments contain only one docbase (DOCBASE1) in addition to the Global Registry. In the DEV environment, there were three docbases for development purposes but the additional two (DOCBASE2 and DOCBASE3) were stopped for a few days because we were running a lot of Performance Tests with EMC and we wanted the results to reflect the TEST/PROD environments. At this point, the Thumbnail Server was working properly, previews were generated successfully, aso… Then to apply a change related to the Performance Tests, we had to restart the whole Content Server, including the Thumbnail Server since it has been installed on all Content Servers (HA environment). The change wasn’t related to the Thumbnail Server at all but we discovered a small bug because of this: after the restart, the Thumbnail Server wasn’t working anymore. Just like the JMS/ACS, there is a way to very quickly know if the Thumbnail Server is up & running or not and that can be checked by entering the following URL in a web browser: http(s)://content_server_01:port/thumbsrv/getThumbnail?

In our case, this URL wasn’t working anymore after the restart while it was working properly before and therefore I had to look at the Thumbnail log file. One important thing to note here is that the Thumbnail Server is bundled with Tomcat. For those of you who are used to work with Tomcat, your first reaction might be to open the file $TOMCAT_HOME/logs/catalina.out. For the Thumbnail, this would be the file: $DM_HOME/thumbsrv/container/logs/catalina.out. That’s what I did… But there were absolutely no useful information because only the Tomcat initialization is displayed in this file by default. The actual useful information for the Thumbnail Server are stored in the localhost log file… That’s the file I’m usually completely ignoring because there is less inside it than in the catalina.out file but apparently EMC took a difference approach!

So let’s take a look at what the log file is providing:

May 04, 2016 2:07:08 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_T_INIT_RESOURCES] Initialized resource strings
May 04, 2016 2:07:08 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Loaded Manifest entries from $DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/WEB-INF/lib/thumbsrv.jar
May 04, 2016 2:07:08 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail:
        Initializing Documentum Thumbnail Server 7.2.0000 , build: 0154
        Created on
        debug             = false
        ticket_timeout    = 300
        application path  = $DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/
May 04, 2016 2:07:08 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Read path to configuration file: $DOCUMENTUM/product/7.2/thumbsrv/conf/user.dat
May 04, 2016 2:07:08 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Initializing Storage Area Manager...
May 04, 2016 2:07:17 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Initializing Connection Manager...
May 04, 2016 2:07:17 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_T_INIT_STORAGE_AREA_MGR] Initialized storage area manager
May 04, 2016 2:07:17 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Initializing crypto classes, key file at $DOCUMENTUM/dba/secure/aek.key
May 04, 2016 2:07:17 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE2
May 04, 2016 2:07:23 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE2
May 04, 2016 2:07:30 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE2
  ...
May 04, 2016 2:10:05 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE2
May 04, 2016 2:10:11 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE2
May 04, 2016 2:10:17 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: initDefaultThumbnailsFromRepository: DOCBASE2 - 0.0
May 04, 2016 2:10:18 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE3
May 04, 2016 2:10:24 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE3
May 04, 2016 2:10:30 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE3
  ...
May 04, 2016 2:13:06 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE3
May 04, 2016 2:13:12 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE3
May 04, 2016 2:13:18 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: initDefaultThumbnailsFromRepository: DOCBASE3 - 0.0
May 04, 2016 2:13:19 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: getRepositoryVersion: DOCBASE1 - 1666666 - 7.2.0050.0214  Linux64.Oracle
May 04, 2016 2:13:19 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: initDefaultThumbnailsFromRepository: DOCBASE1 - 7.2
May 04, 2016 2:13:20 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: createDefaultThumnailsInRespoitory: repo=/System/ThumbnailServer/thumbnails, local=$DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/thumbnails
May 04, 2016 2:13:22 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: createDefaultThumnailsInRespoitory: repo=/System/ThumbnailServer/thumbnails/formats, local=$DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/thumbnails/formats
May 04, 2016 2:13:22 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: createDefaultThumnailsInRespoitory: repo=/System/ThumbnailServer/thumbnails/types, local=$DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/thumbnails/types
May 04, 2016 2:13:26 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: checkRuleVersion: the default thumbnail rule version - 6.0.0.101
May 04, 2016 2:13:26 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: initRepositoryRules: repoitory=DOCBASE1, id=1666666
May 04, 2016 2:13:34 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_T_INIT_DEF_THUMB_MGR] Initialized default thumbnails manager
May 04, 2016 2:13:34 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Failed to get a session for DOCBASE2: DfNoServersException:: THREAD: pool-2-thread-1; MSG: [DM_DOCBROKER_E_NO_SERVERS_FOR_DOCBASE]error:  "The DocBroker running on host (content_server_01:1489) does not know of a server for the specified docbase (DOCBASE2)"; ERRORCODE: 100; NEXT: null
May 04, 2016 2:13:34 PM org.apache.catalina.core.ApplicationContext log
SEVERE: getThumbnail: [DM_TS_E_INIT_FORMATS_MGR] Falied to initialize formats.
DfNoServersException:: THREAD: pool-2-thread-1; MSG: [DM_DOCBROKER_E_NO_SERVERS_FOR_DOCBASE]error:  "The DocBroker running on host (content_server_01:1489) does not know of a server for the specified docbase (DOCBASE2)"; ERRORCODE: 100; NEXT: null
        at com.documentum.fc.client.impl.docbroker.ServerMapBuilder.__AW_getDataFromDocbroker(ServerMapBuilder.java:171)
        at com.documentum.fc.client.impl.docbroker.ServerMapBuilder.getDataFromDocbroker(ServerMapBuilder.java)
        at com.documentum.fc.client.impl.docbroker.ServerMapBuilder.__AW_getMap(ServerMapBuilder.java:60)
        at com.documentum.fc.client.impl.docbroker.ServerMapBuilder.getMap(ServerMapBuilder.java)
        at com.documentum.fc.client.impl.docbroker.DocbrokerClient.getServerMap(DocbrokerClient.java:152)
        at com.documentum.fc.client.impl.connection.docbase.ServerChoiceManager.__AW_updateServerChoices(ServerChoiceManager.java:159)
        at com.documentum.fc.client.impl.connection.docbase.ServerChoiceManager.updateServerChoices(ServerChoiceManager.java)
        at com.documentum.fc.client.impl.connection.docbase.ServerChoiceManager.updateServerChoicesIfNecessary(ServerChoiceManager.java:148)
        at com.documentum.fc.client.impl.connection.docbase.ServerChoiceManager.getServerChoices(ServerChoiceManager.java:47)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.getServerChoices(DocbaseConnection.java:273)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.__AW_establishNewRpcClient(DocbaseConnection.java:227)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.establishNewRpcClient(DocbaseConnection.java)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.__AW_open(DocbaseConnection.java:126)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.open(DocbaseConnection.java)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.<init>(DocbaseConnection.java:100)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.<init>(DocbaseConnection.java:60)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionFactory.newDocbaseConnection(DocbaseConnectionFactory.java:26)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionManager.createNewConnection(DocbaseConnectionManager.java:180)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionManager.getDocbaseConnection(DocbaseConnectionManager.java:110)
        at com.documentum.fc.client.impl.session.SessionFactory.newSession(SessionFactory.java:23)
        at com.documentum.fc.client.impl.session.PrincipalAwareSessionFactory.newSession(PrincipalAwareSessionFactory.java:44)
        at com.documentum.fc.client.impl.session.PooledSessionFactory.__AW_newSession(PooledSessionFactory.java:49)
        at com.documentum.fc.client.impl.session.PooledSessionFactory.newSession(PooledSessionFactory.java)
        at com.documentum.fc.client.impl.session.SessionManager.getSessionFromFactory(SessionManager.java:134)
        at com.documentum.fc.client.impl.session.SessionManager.newSession(SessionManager.java:72)
        at com.documentum.fc.client.impl.session.SessionManager.getSession(SessionManager.java:191)
        at com.documentum.thumbsrv.docbase.DocbaseConnectionMgr.getSessionForSection(DocbaseConnectionMgr.java:197)
        at com.documentum.thumbsrv.docbase.FormatMapperMgr.__AW_loadFormatsFromRepositories(FormatMapperMgr.java:95)
        at com.documentum.thumbsrv.docbase.FormatMapperMgr.loadFormatsFromRepositories(FormatMapperMgr.java)
        at com.documentum.thumbsrv.docbase.FormatMapperMgr.<init>(FormatMapperMgr.java:62)
        at com.documentum.thumbsrv.getThumbnail.__AW_init(getThumbnail.java:214)
        at com.documentum.thumbsrv.getThumbnail.init(getThumbnail.java)
        at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1266)
        at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1185)
        at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1080)
        at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5015)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5302)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:895)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:871)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:615)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1095)
        at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1617)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.__AW_run(FutureTask.java:262)
        at java.util.concurrent.FutureTask.run(FutureTask.java)
        at java.util.concurrent.ThreadPoolExecutor.__AW_runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
 
May 04, 2016 2:13:34 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_E_INIT_FAILED] Failed to initialize Documentum Thumbnail Server
May 04, 2016 2:13:34 PM org.apache.catalina.core.ApplicationContext log
INFO: Marking servlet getThumbnail as unavailable
May 04, 2016 2:13:34 PM org.apache.catalina.core.StandardContext loadOnStartup
SEVERE: Servlet /thumbsrv threw load() exception
javax.servlet.UnavailableException: [DM_TS_E_INIT_FAILED] Failed to initialize Documentum Thumbnail Server
        at com.documentum.thumbsrv.getThumbnail.__AW_init(getThumbnail.java:221)
        at com.documentum.thumbsrv.getThumbnail.init(getThumbnail.java)
        at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1266)
        at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1185)
        at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1080)
        at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5015)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5302)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:895)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:871)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:615)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1095)
        at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1617)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.__AW_run(FutureTask.java:262)
        at java.util.concurrent.FutureTask.run(FutureTask.java)
        at java.util.concurrent.ThreadPoolExecutor.__AW_runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
 

May 04, 2016 2:17:16 PM org.apache.catalina.core.StandardWrapperValve invoke
INFO: Servlet getThumbnail is currently unavailable
May 04, 2016 2:22:16 PM org.apache.catalina.core.StandardWrapperValve invoke
INFO: Servlet getThumbnail is currently unavailable
May 04, 2016 2:27:16 PM org.apache.catalina.core.StandardWrapperValve invoke
INFO: Servlet getThumbnail is currently unavailable
May 04, 2016 2:32:16 PM org.apache.catalina.core.StandardWrapperValve invoke
INFO: Servlet getThumbnail is currently unavailable
May 04, 2016 2:37:16 PM org.apache.catalina.core.StandardWrapperValve invoke
INFO: Servlet getThumbnail is currently unavailable
May 04, 2016 2:42:16 PM org.apache.catalina.core.StandardWrapperValve invoke
INFO: Servlet getThumbnail is currently unavailable
May 04, 2016 2:47:16 PM org.apache.catalina.core.StandardWrapperValve invoke
INFO: Servlet getThumbnail is currently unavailable

Ok that’s quite a long trace and I already cut some parts because it was really too long. As you can see above, the very beginning of the startup is going on properly and then the Thumbnail Server is trying to contact the DOCBASE2. Hum why exactly is it trying to contact the DOCBASE2 while this docbase isn’t running? That’s a first strange thing. Then I cut a lot of lines but you can see that it is actually trying to do that for 3 minutes and it is not doing anything else during that time. Once the three minutes are over, it is trying to contact the DOCBASE3 for 3 minutes too (and failed again) and finally it contacted the DOCBASE1 which was the only docbase running at that time. This last one succeeded so the Thumbnail Server should have started properly… But actually it couldn’t because the first docbase that it tried fail (DOCBASE2) and that’s the docbase that will be used by the Thumbnail Server to open a session to retrieve some information.

So I did more tests to try to understand where the issue was and what could be done to solve that. First of all, the important thing to understand here is that the Thumbnail Server isn’t trying to contact all docbases ever installed on this Content Server. It will only try to contact the docbases that have been configured for it. To be more precise, the configuration of a docbase for the Thumbnail Server will update the file user.dat ($DM_HOME/thumbsrv/conf/user.dat) and add inside it the configuration for this specific docbase. When the Thumbnail Server will start, it will parse this file and see which docbases should be contacted during the startup.

Therefore my first test was to simply comment all lines related to DOCBASE2 and DOCBASE3 and then restart the Thumbnail Server. This is what I got:

May 04, 2016 3:23:32 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_T_INIT_RESOURCES] Initialized resource strings
May 04, 2016 3:23:32 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Loaded Manifest entries from $DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/WEB-INF/lib/thumbsrv.jar
May 04, 2016 3:23:32 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail:
        Initializing Documentum Thumbnail Server 7.2.0000 , build: 0154
        Created on
        debug             = false
        ticket_timeout    = 300
        application path  = $DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/
May 04, 2016 3:23:32 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Read path to configuration file: $DOCUMENTUM/product/7.2/thumbsrv/conf/user.dat
May 04, 2016 3:23:32 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Initializing Storage Area Manager...
May 04, 2016 3:23:40 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Initializing Connection Manager...
May 04, 2016 3:23:40 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_T_INIT_STORAGE_AREA_MGR] Initialized storage area manager
May 04, 2016 3:23:40 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Initializing crypto classes, key file at $DOCUMENTUM/dba/secure/aek.key
May 04, 2016 3:23:40 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: getRepositoryVersion: DOCBASE1 - 1666666 - 7.2.0050.0214  Linux64.Oracle
May 04, 2016 3:23:40 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: initDefaultThumbnailsFromRepository: DOCBASE1 - 7.2
May 04, 2016 3:23:41 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: createDefaultThumnailsInRespoitory: repo=/System/ThumbnailServer/thumbnails, local=$DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/thumbnails
May 04, 2016 3:23:43 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: createDefaultThumnailsInRespoitory: repo=/System/ThumbnailServer/thumbnails/formats, local=$DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/thumbnails/formats
May 04, 2016 3:23:43 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: createDefaultThumnailsInRespoitory: repo=/System/ThumbnailServer/thumbnails/types, local=$DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/thumbnails/types
May 04, 2016 3:23:47 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: checkRuleVersion: the default thumbnail rule version - 6.0.0.101
May 04, 2016 3:23:47 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: initRepositoryRules: repoitory=DOCBASE1, id=1666666
May 04, 2016 3:23:54 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_T_INIT_DEF_THUMB_MGR] Initialized default thumbnails manager
May 04, 2016 3:23:54 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_T_INIT_COMPLETE] Documentum Thumbnail Server initialization complete!

As you can see, the content is exactly the same except that the Thumbnail Server is now only contacting the DOCBASE1 (successfully again) but this time, the Thumbnail Server is up & running properly. Therefore commenting the lines in the file user.dat solved this issue… But that’s not enough. For me, this was clearly a bug and therefore I did more tests.

For the next test, I restored the lines related to DOCBASE2 and DOCBASE3 in the file user.dat and I tried to change the order of the lines inside this file… Because DOCBASE1 was the first docbase in this file, DOCBASE2 the second and DOCBASE3 the last one. So I thought that maybe the second docbase was the first to be initialized? And if the first docbase to be initialized (2nd in the file?) is responding, will the Thumbnail Server work?

Therefore I changed the order:

• Switching DOCBASE1 in second position, DOCBASE2 in first position, DOCBASE3 still in last => Not working, same issue, DOCBASE2 is the first to be initialised
• Switching DOCBASE1 in second position, DOCBASE2 in last position, DOCBASE3 in first position => Not working, same issue, DOCBASE3 is the first to be initialised
• …

This test wasn’t successful because there is absolutely no logic behind which docbase will be initialized first… Therefore I restored the initial values (1 in first, 2 in second, 3 in last) and I performed a last test: stopping the DOCBASE1 and starting the DOCBASE2 so that there is still one docbase to be running which would be the first to be initialized. After doing that, I restarted a last time the Thumbnail Server and the first docbase to be initialized was indeed the DOCBASE2, which was running. Then it tried to initialise the DOCBASE1 and DOCBASE3 which weren’t running and therefore it failed. BUT in the end, the Thumbnail Server was able to start properly, even if it took 6 minutes + ~20 seconds instead of 20 seconds to start.

With all these results, I opened a Service Request on the EMC website and they were able to find the root cause for both issues (why it is looping for 3 minutes on nothing and why it is sometimes not able to start when one docbase is properly responding). In the end, they provided us a hotfix that has been incorporated in newer versions of the software normally and this hotfix fixed both issues properly.