At one of our customer, we were in the middle of a migration process of some docbases from 6.7 to 7.2. A few days after the migration, we started seeing some failures/errors during simple download of documents from D2 4.5. The migration has been done using the EMC EMA migration tool by some EMC colleagues. The strange thing here is that these download failures only applied to a few documents, far from the majority and only when opening the document using “View Native Content”. In addition to that, it appeared that the issue was only on migrated documents and it wasn’t happening for new ones.

 

This is an example of the error message we were able to see in the D2 4.5 log files:

2016-07-04 12:00:20 [ERROR] [[ACTIVE] ExecuteThread: '326' for queue: 'weblogic.kernel.Default (self-tuning)'] - c.e.d.d.s.D2HttpServlet[                    ] : Download failed
java.net.ProtocolException: Exceeded stated content-length of: '63000' bytes
        at weblogic.servlet.internal.ServletOutputStreamImpl.checkCL(ServletOutputStreamImpl.java:217) [weblogic.server.merged.jar:12.1.3.0.0]
        at weblogic.servlet.internal.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:162) [weblogic.server.merged.jar:12.1.3.0.0]
        at com.emc.d2fs.dctm.servlets.ServletUtil.download(ServletUtil.java:375) [D2FS4DCTM-API-4.5.0.jar:na]
        at com.emc.d2fs.dctm.servlets.ServletUtil.download(ServletUtil.java:280) [D2FS4DCTM-API-4.5.0.jar:na]
        at com.emc.d2fs.dctm.servlets.download.Download.processRequest(Download.java:132) [D2FS4DCTM-WEB-4.5.0.jar:na]
        at com.emc.d2fs.dctm.servlets.D2HttpServlet.execute(D2HttpServlet.java:242) [D2FS4DCTM-API-4.5.0.jar:na]
        at com.emc.d2fs.dctm.servlets.D2HttpServlet.doGetAndPost(D2HttpServlet.java:498) [D2FS4DCTM-API-4.5.0.jar:na]
        at com.emc.d2fs.dctm.servlets.D2HttpServlet.doGet(D2HttpServlet.java:115) [D2FS4DCTM-API-4.5.0.jar:na]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) [weblogic.server.merged.jar:12.1.3.0.0]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:844) [weblogic.server.merged.jar:12.1.3.0.0]
        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:280) [weblogic.server.merged.jar:12.1.3.0.0]
        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:254) [weblogic.server.merged.jar:12.1.3.0.0]
        at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:136) [weblogic.server.merged.jar:12.1.3.0.0]
        at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:346) [weblogic.server.merged.jar:12.1.3.0.0]
        at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25) [weblogic.server.merged.jar:12.1.3.0.0]
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79) [weblogic.server.merged.jar:12.1.3.0.0]
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79) [weblogic.server.merged.jar:12.1.3.0.0]
        at com.emc.x3.portal.server.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:77) [_wl_cls_gen.jar:na]
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79) [weblogic.server.merged.jar:12.1.3.0.0]
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66) [guice-servlet-3.0.jar:na]
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) [shiro-web-1.1.0.jar:na]
        at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) [shiro-web-1.1.0.jar:na]
        at com.company.d2.auth.NonSSOAuthenticationFilter.executeChain(NonSSOAuthenticationFilter.java:21) [_wl_cls_gen.jar:na]
        at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) [shiro-web-1.1.0.jar:na]
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:81) [shiro-web-1.1.0.jar:na]
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) [shiro-web-1.1.0.jar:na]
        at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:359) [shiro-web-1.1.0.jar:na]
        at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:275) [shiro-web-1.1.0.jar:na]
        at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) [shiro-core-1.1.0.jar:1.1.0]
        at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) [shiro-core-1.1.0.jar:1.1.0]
        at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:344) [shiro-core-1.1.0.jar:1.1.0]
        at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:272) [shiro-web-1.1.0.jar:na]
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:81) [shiro-web-1.1.0.jar:na]
        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) [guice-servlet-3.0.jar:na]
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) [guice-servlet-3.0.jar:na]
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
        at com.planetj.servlet.filter.compression.CompressingFilter.doFilter(CompressingFilter.java:270) [pjl-comp-filter-1.7.jar:na]
        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) [guice-servlet-3.0.jar:na]
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) [guice-servlet-3.0.jar:na]
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) [guice-servlet-3.0.jar:na]
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79) [weblogic.server.merged.jar:12.1.3.0.0]
        at com.emc.x3.portal.server.filters.X3SessionTimeoutFilter.doFilter(X3SessionTimeoutFilter.java:34) [_wl_cls_gen.jar:na]
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79) [weblogic.server.merged.jar:12.1.3.0.0]
        at com.planetj.servlet.filter.compression.CompressingFilter.doFilter(CompressingFilter.java:270) [pjl-comp-filter-1.7.jar:na]
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79) [weblogic.server.merged.jar:12.1.3.0.0]
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3436) [weblogic.server.merged.jar:12.1.3.0.0]
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3402) [weblogic.server.merged.jar:12.1.3.0.0]
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) [weblogic.server.merged.jar:12.1.3.0.0]
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120) [com.oracle.css.weblogic.security.wls_7.1.0.0.jar:CSS 7.1 0.0]
        at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57) [weblogic.server.merged.jar:12.1.3.0.0]
        at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2285) [weblogic.server.merged.jar:12.1.3.0.0]
        at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2201) [weblogic.server.merged.jar:12.1.3.0.0]
        at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179) [weblogic.server.merged.jar:12.1.3.0.0]
        at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1572) [weblogic.server.merged.jar:12.1.3.0.0]
        at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:255) [weblogic.server.merged.jar:12.1.3.0.0]
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311) [weblogic.server.merged.jar:12.1.3.0.0]
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:263) [weblogic.server.merged.jar:12.1.3.0.0]

 

So the error in this case is “Exceeded stated content-length of: ‘63000’ bytes”. Hum what does that mean? Well it is not really clear (who said not clear at all?)… So we checked several documents for which the download failed (iAPI using dumps) and the only common points we were able to find for these documents were the following ones:

  • They all had a r_full_content_size of: 0
  • They all had a r_content_size bigger than: 63 000

 

The issue only appeared for objects which were assigned a r_full_content_size of 0 during the migration. We tried to set this property to 0 on a document for which the download was working, in order to try to reproduce the issue, but that didn’t change anything: the download was still working properly.

 

So here is some background regarding this parameter: the expected behavior for this parameter is that it has a real value (obviously). If the file in question is smaller than 2 GB, then the r_full_content_size will have the same value as the r_content_size which is of course the size of the file in bytes. If the file is bigger than 2 GB, then the r_content_size field is too small to show the real size in bytes and therefore the real size is only displayed in the r_full_content_size field… The r_full_content_size is the one read by D2 when using the “View Native Content” action while other actions like “View” or “Edit” behave like older wdk clients and are therefore reading the r_content_size attribute… So yes there is a difference in behavior between the few actions that are doing a download and that’s the reason why we only had this issue with the “View Native Content” action!

 

Unfortunately and as you probably already understood if you read the previous paragraph, the r_content_size and r_full_content_size aren’t of the same type (Integer Vs. Double) and therefore you can’t simply execute one single DQL to set the value of r_full_content_size equal to the value of r_content_size because you will get a DM_QUERY_E_UP_BAD_ATTR_TYPES error. So you will have to do things a little bit slower.

 

The first thing to do is obviously to gather a list of all documents that need to be updated with their r_object_id and value of their r_content_size (r_full_content_size don’t really matter since you will gather only affected documents so this value is always 0):

> SELECT r_object_id, r_content_size FROM dm_document WHERE r_full_content_size='0' AND r_content_size>'63000';
r_object_id         r_content_size
------------------- -------------------
090f446780034513    289326
090f446780034534    225602
090f446780034536    212700
090f446780034540    336916
090f446780034559    269019
090f446780034572    196252
090f446780034574    261094
090f44678003459a    232887
...                 ...

 

Then a first solution would be to manually go through the list and execute one DQL query for each document setting r_full_content_size=’241309′ (or whatever the r_content_size is). For the first document listed above, that would therefore be the following DQL:

> UPDATE dm_document objects SET r_full_content_size='289326' WHERE r_object_id='090f446780034513';

 

Note 1: In case you will want at some point to restore a previous version of a document for example, then you will most probably need to use the “FROM dm_document(ALL)” instead of “FROM dm_document”. But then take care that non current versions are immutable and can’t therefore be updated using a simple DQL. You will need to remove the immutable flag, update the object and restore that so that’s a little bit more tricky 😉

Note 2: In case you have a few documents bigger than 2 GB, the r_content_size will not reflect the real value and therefore setting the r_full_content_size to that value isn’t correct… I wasn’t able to test that since our customer didn’t have any document bigger than 2 GB but you should most probably be able to use instead the full_content_size that is stored on the dmr_content for this object… Just like the dm_document, the dmr_content object has two fields that you should be able to use to find the correct size: content_size (that should reflect r_content_size) and full_content_size (that should reflect r_full_content_size). If that isn’t helping then a last solution would be to export and re-import all documents bigger than 2 GB…

 

Ok so updating all objects is possible but this is really boring so a second solution – and probably a better one – is to use a script to prepare a list of DQL queries to be executed. When you have the r_object_id and r_content_size of all affected documents, you can just put that in a CSV file (copy/paste in excel and save as CSV for example) and wrote a small script (bash for example) that will generate 1 DQL query per document, that’s really simple and if you have thousands of documents affected, then it will just take you a few minutes to write the script and in 1/2 seconds you will have thousands of DQL queries generated. Then you can put all these commands in a single file that can be executed against a docbase on the Content Server. That’s a better solution but actually the simplest solution you can ever find will always be to use dqMan (or any similar solution). Indeed dqMan has a dedicated feature that allows you to execute a “template DQL” on any list of objects returned by a specific command. Therefore you don’t need any bash scripting if you are using dqMan and that does the job in a few seconds.

 

A last solution would be to go directly to the database and execute SQL queries to set r_full_content_size equal to r_content_size BUT I would NOT recommend you to do that unless you have a very good knowledge of the Documentum Content Model and if you absolutely know what you are doing and what you are messing with ;).

 

See you soon!