A few months ago and right after a migration of around 0.5TB of documents, we enabled the TCS for one of our Application. We were using a Content Server 7.2 P05 with the associated D2 4.5 P03. As already mentioned in a previous blog, D2 4.5 doesn’t handle the previews of the documents and therefore there were also a Thumbnail Server used by this Application. The setup of the TCS for the document filestores went well without issue but when we tried to do the same thing for the Thumbnail filestore, the previews weren’t working anymore.
Basically when you configure the Thumbnail to use a TCS filestore, you need to request new renditions for existing documents otherwise they will continue to use the non-TCS filestore.
If you access D2 while inspecting the network traffic or using the browser dev/debug feature, you will find that D2 is building the Thumbnail preview URL using the value of the “dm_document.thumbnail_url”. This thumbnail_url is actually – by default – a concatenation of several things: thumbnail_url = base_url + path + store.
Therefore if you define:
- the Thumbnail base_url to be “https://thumbnail_alias/thumbsrv/getThumbnail?”
- the Thumbnail TCS filestore to be “trusted_thumbnail_store_01”
- a test document 09012345801a8f56 with a file system path (get_path) to be: “00012345/80/00/10/4d.jpg”
Then D2 will retrieve a thumbnail_url that is:
https://thumbnail_alias/thumbsrv/getThumbnail?path=00012345/80/00/10/4d.jpg&store=trusted_thumbnail_store_01
When accessing this URL, if the filestore mentioned above is indeed a TCS filestore, this is the kind of logs that will be generated:
Jul 05, 2016 9:42:15 AM org.apache.catalina.core.ApplicationContext log INFO: getThumbnail: [DEBUG] : [DM_TS_T_RETRIEVE_DOCBASE_THUMB] Retrieving docbase thumbnail... Jul 05, 2016 9:42:15 AM org.apache.catalina.core.ApplicationContext log INFO: getThumbnail: [DEBUG] : the ETag received is the path - will return 304 response Jul 05, 2016 9:45:12 AM org.apache.catalina.core.ApplicationContext log INFO: getThumbnail: [DEBUG] : [DM_TS_T_NEW_REQUEST] New request: path=00012345/80/00/10/4d.jpg&store=trusted_thumbnail_store_01 Jul 05, 2016 9:45:12 AM org.apache.catalina.core.ApplicationContext log INFO: getThumbnail: [DEBUG] : [DM_TS_T_RETRIEVE_DOCBASE_THUMB] Retrieving docbase thumbnail... Jul 05, 2016 9:45:12 AM org.apache.catalina.core.ApplicationContext log INFO: getThumbnail: [DEBUG] : [DM_TS_T_RETRIEVE_STORE] Retrieving storage area... Jul 05, 2016 9:45:12 AM org.apache.catalina.core.ApplicationContext log INFO: getThumbnail: [DEBUG] : getting storage area for docbase: 00012345 store: trusted_thumbnail_store_01 Jul 05, 2016 9:45:12 AM org.apache.catalina.core.ApplicationContext log INFO: getThumbnail: [DEBUG] : [DM_TS_T_CHECK_SECURITY] Checking security... Jul 05, 2016 9:45:12 AM org.apache.catalina.core.ApplicationContext log INFO: getThumbnail: [DEBUG] : About to start reading files... Jul 05, 2016 9:45:12 AM org.apache.catalina.core.ApplicationContext log INFO: getThumbnail: [DEBUG] : Trusted content store found, will use DFC for decryption Jul 05, 2016 9:45:12 AM org.apache.catalina.core.ApplicationContext log INFO: getThumbnail: [DEBUG] : Content paramters are: format= null; page=null; page modifier=null Jul 05, 2016 9:45:12 AM org.apache.catalina.core.ApplicationContext log INFO: getThumbnail: Object ID missing from request...
When using a non-TCS filestore, this url will retrieve the document preview properly (of course the name of the store was “thumbnail_store_01” and not “trusted_thumbnail_store_01”) but as you can see above, with a TCS filestore, the Thumbnail is trying to use the DFC for decryption but isn’t able to because the Object ID is missing on the request… With a non-TCS filestore, the Thumbnail is retrieving the content directly from the file system but it’s not possible to do that with a TCS filestore because of the encryption. Therefore a TCS-enabled Thumbnail has to use the “getContent” method of the DFC and this method requires the Object ID. That’s the issue here.
When we faced this issue last summer, we contacted EMC because that was clearly a bug in how D2 is constructing the URLs to request the preview and not how the Thumbnail Server is processing the requested URLs. After several days, the D2 engineering team provided a hotfix to enable D2 to provide the correct parameters. Basically with the new hotfix deployed, D2 was now able to generate the following URL:
https://thumbnail_alias/thumbsrv/getThumbnail?path=00012345/80/00/10/4d.jpg&store=trusted_thumbnail_store_01&did=09012345801a8f56&format=jpeg_lres
Note: In this case, we also had to specify the format (jpeg_lres). While waiting for the EMC hotfix, we performed some tests on our side by trying to add missing parameters and trying to understand what is needed for the Thumbnail Server in the requests. At some point in time, we found out that the Thumbnail was trying to use “jpeg_th” as a default value for the format if you don’t specify any. As far as I know, we have a pretty default Thumbnail configuration but we don’t have any “jpeg_th” formats. Only jpeg_lres and jpeg_story are used and therefore I don’t know where this jpeg_th is coming from. I believe that’s an issue of the Thumbnail Server because EMC included the format in the D2 hotfix too, after we mentioned that to them.
So using this new URL, the generated log files were:
Jul 12, 2016 6:29:53 AM org.apache.catalina.core.ApplicationContext log INFO: getThumbnail: [DEBUG] : [DM_TS_T_NEW_REQUEST] New request: path=00012345/80/00/10/4d.jpg&store=trusted_thumbnail_store_01&did=09012345801a8f56&format=jpeg_lres Jul 12, 2016 6:29:53 AM org.apache.catalina.core.ApplicationContext log INFO: getThumbnail: [DEBUG] : [DM_TS_T_RETRIEVE_DOCBASE_THUMB] Retrieving docbase thumbnail... Jul 12, 2016 6:29:53 AM org.apache.catalina.core.ApplicationContext log INFO: getThumbnail: [DEBUG] : [DM_TS_T_RETRIEVE_STORE] Retrieving storage area... Jul 12, 2016 6:29:53 AM org.apache.catalina.core.ApplicationContext log INFO: getThumbnail: [DEBUG] : getting storage area for docbase: 00012345 store: trusted_thumbnail_store_01 Jul 12, 2016 6:29:53 AM org.apache.catalina.core.ApplicationContext log INFO: getThumbnail: [DEBUG] : [DM_TS_T_CHECK_SECURITY] Checking security... Jul 12, 2016 6:29:53 AM org.apache.catalina.core.ApplicationContext log INFO: getThumbnail: [DEBUG] : About to start reading files... Jul 12, 2016 6:29:53 AM org.apache.catalina.core.ApplicationContext log INFO: getThumbnail: [DEBUG] : Trusted content store found, will use DFC for decryption Jul 12, 2016 6:29:53 AM org.apache.catalina.core.ApplicationContext log INFO: getThumbnail: [DEBUG] : Content paramters are: format= jpeg_lres; page=null; page modifier=null Jul 12, 2016 6:29:53 AM org.apache.catalina.core.ApplicationContext log INFO: getThumbnail: [DEBUG] : will be retrieving object 09012345801a8f56 Jul 12, 2016 6:29:53 AM org.apache.catalina.core.ApplicationContext log INFO: getThumbnail: [DEBUG] : Session ok for 00012345
And it worked; the preview of the first page of this document has been displayed. The hotfix seemed to work so we started to deploy it in our DEV environments while waiting for a confirmation from the Application Team that it was also working from D2 (because of SoD, our admin accounts don’t have access to documents in D2). But then after a few tests, The App Team found out that the hotfix was actually only partially working: only for the first page! Indeed EMC created this hotfix using a one-page document and they never tried to retrieve the previews of a multi-page document. The thing with the Thumbnail Server is that if you have a 25-pages document, then this document will have 25 previews. I already talked about that in another previous blog so you can take a look at this blog to have more information on how to manage that.
I will suppose that 09012345801a8f56 is a 3-pages document. I gathered some information from this document and also got the path of the thumbnail previews that are related to the 3 pages:
API> ?,c,select r_object_id, full_format, parent_id, content_size, full_content_size, set_time, set_file from dmr_content where any parent_id='09012345801a8f56' r_object_id full_format parent_id content_size full_content_size set_time set_file ---------------- ----------- ---------------- ------------ ----------------- ----------------- -------------------------------------------------------------------- 06012345801d95cc jpeg_lres 09012345801a8f56 60467 60467 7/4/2016 13:22:39 C:UsersSYS_AD~1AppDataLocalTempbatchFile353776780929575961.tar get_path => /data/DOCBASE/trusted_thumbnail_storage_01/00012345/80/00/10/4d.jpg 06012345801d95cd jpeg_lres 09012345801a8f56 138862 138862 7/4/2016 13:22:39 C:UsersSYS_AD~1AppDataLocalTempbatchFile353776780929575961.tar get_path => /data/DOCBASE/trusted_thumbnail_storage_01/00012345/80/00/10/4e.jpg 06012345801d95ce jpeg_lres 09012345801a8f56 29596 29596 7/4/2016 13:22:39 C:UsersSYS_AD~1AppDataLocalTempbatchFile353776780929575961.tar get_path => /data/DOCBASE/trusted_thumbnail_storage_01/00012345/80/00/10/4f.jpg
So here we have three different jpg files for the same “parent_id” (the document) and each one is a preview of a specific page. These previews have a different size (content_size/full_content_size), therefore the previews should be different!
With the information provided above, the previews URLs generated by D2 (using the new hotfix) are:
https://thumbnail_alias/thumbsrv/getThumbnail?path=00012345/80/00/10/4d.jpg&store=trusted_thumbnail_store_01&did=09012345801a8f56&format=jpeg_lres https://thumbnail_alias/thumbsrv/getThumbnail?path=00012345/80/00/10/4e.jpg&store=trusted_thumbnail_store_01&did=09012345801a8f56&format=jpeg_lres https://thumbnail_alias/thumbsrv/getThumbnail?path=00012345/80/00/10/4f.jpg&store=trusted_thumbnail_store_01&did=09012345801a8f56&format=jpeg_lres
Three different URLs for three different pages. That looks fine but actually if you access all these three URLs, you will always see the preview of the first page of the document… It goes even beyond that! Actually you can put any path in the URL and it will always show the first page of the document (09012345801a8f56). To confirm that, you can access the following URLs and see by yourself that they will also display the same preview:
https://thumbnail_alias/thumbsrv/getThumbnail?path=00012345/80/00/10/zz.jpg&store=trusted_thumbnail_store_01&did=09012345801a8f56&format=jpeg_lres https://thumbnail_alias/thumbsrv/getThumbnail?path=00012345/zz/zz/zz/zz.jpg&store=trusted_thumbnail_store_01&did=09012345801a8f56&format=jpeg_lres https://thumbnail_alias/thumbsrv/getThumbnail?path=00012345/&store=trusted_thumbnail_store_01&did=09012345801a8f56&format=jpeg_lres
Based on all the above information, you can understand that the way the Thumbnail Server is working is really different when using a non-TCS or a TCS filestore… For non-TCS filestores, the URL points directly to the actual file and that’s the reason why the full path is really important: it leads directly to the right file and each page has its own path. For TCS filestores, since all the files are encrypted, the Thumbnail Server can’t directly use the file path. Therefore it relies on the DFC to first decrypt the file and then return the result. That’s the reason why only the docbase ID is needed inside the “path” when using a TCS and that everything else in the path is completely useless. On the other end and as seen previously in this blog, you of course also need some additional information to specify which preview you want.
In addition to the parameters we saw above and in order to uniquely identify different renditions, page and/or page_modifier is also required with a TCS filestore. Both attributes are part of dmr_content table and page is the position of the content when the object has multiple contents (generally it will always be zero by default) while page_modifier uniquely identifies a rendition within the same page number and format for a document. If no/null page_modifier is passed in the URL then the rendition with an empty page_modifier value is returned. If there isn’t any rendition in the docbase without a page_modifier value, then the one with the smallest page_modifier (in alphabetical order) will be returned instead: that’s the reason why all previous URLs always returned the preview of the first page only… In short, for non-TCS filestores, the path pretty much does the job, but for TCS filestores we need additional parameters to uniquely identify the rendition.
So to summarize the investigation we did with EMC (because yes we found a definitive solution), the minimum elements that should be present in the Thumbnail URLs are:
- non-TCS filestore:
- Full path to the right file
- Store used
- TCS filestore:
- Path containing at least the Docbase ID
- Store used
- Document ID (parent_id)
- Format of the preview
- page and/or page_modifier (from the dmr_content table)
In the end, EMC provided another hotfix which fix the first hotfix. The complete URLs are now generated by D2 and the previews are working properly even with a TCS Thumbnail filestore. Here are the examples for the 3-pages document:
https://thumbnail_alias/thumbsrv/getThumbnail?path=00012345/80/00/10/4d.jpg&store=trusted_thumbnail_store_01&did=09012345801a8f56&format=jpeg_lres&page_modifier=000000001 https://thumbnail_alias/thumbsrv/getThumbnail?path=00012345/80/00/10/4e.jpg&store=trusted_thumbnail_store_01&did=09012345801a8f56&format=jpeg_lres&page_modifier=000000002 https://thumbnail_alias/thumbsrv/getThumbnail?path=00012345/80/00/10/4f.jpg&store=trusted_thumbnail_store_01&did=09012345801a8f56&format=jpeg_lres&page_modifier=000000003
The definitive hotfix should normally be included in all the 2017 releases according to EMC.
![Thumbnail [60x60]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/MOP_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/GME_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/ATR_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/CVI_web-min-scaled.jpg)