At a customer, the D2-REST (16.5.1) application hosted on WebLogic Server 12c started showing 500 Internal Server Errors, after a customer release including many things. The error was rather simple to replicate since opening the D2-REST Product info page was sufficient (https://<host>/D2-REST/product-info). The URL was returning the following:
At the same time, on the logs:
2021-04-26 06:46:20,340 UTC [ERROR] ([ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)') - com.emc.documentum.rest.util.LogHelper : LogId: 9b360f83-335a-413e-87e3-481ba5cbf168, Status: 500, code: E_INTERNAL_SERVER_ERROR, message: An internal server error occurs.
org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.NoSuchMethodError: com.google.common.base.Objects.firstNonNull(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:982)
at com.emc.documentum.rest.servlet.RestDispatcherServlet.doDispatch(RestDispatcherServlet.java:33)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:286)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:350)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at com.emc.documentum.rest.filter.ApplicationFilter.doFilter(ApplicationFilter.java:33)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at com.emc.documentum.d2.rest.filter.AppValidationFilter.doFilter(AppValidationFilter.java:35)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at com.emc.documentum.d2.rest.filter.AppInfoFilter.doFilter(AppInfoFilter.java:39)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at com.emc.documentum.rest.security.filter.RepositoryNamingFilter.doFilter(RepositoryNamingFilter.java:40)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at com.emc.documentum.rest.filter.RestCorsFilter.doFilterInternal(RestCorsFilter.java:47)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at com.emc.documentum.rest.filter.CompressionFilter.doFilter(CompressionFilter.java:73)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at com.emc.documentum.rest.log.MessageLoggingFilter.doFilter(MessageLoggingFilter.java:69)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at com.emc.documentum.rest.security.filter.ExceptionHandlerFilter.doFilterInternal(ExceptionHandlerFilter.java:31)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3706)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3672)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:344)
at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197)
at weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203)
at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71)
at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2443)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2291)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2269)
at weblogic.servlet.internal.ServletRequestImpl.runInternal(ServletRequestImpl.java:1705)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1665)
at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:272)
at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:652)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:420)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:360)
Caused by: java.lang.NoSuchMethodError: com.google.common.base.Objects.firstNonNull(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
at com.emc.documentum.d2fs.controller.D2AppInfoController.attribute(D2AppInfoController.java:160)
at com.emc.documentum.d2fs.controller.D2AppInfoController.getProductInfo(D2AppInfoController.java:94)
at com.emc.documentum.d2fs.controller.D2AppInfoController.get(D2AppInfoController.java:65)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:849)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:760)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
... 72 common frames omitted
2021-04-26 06:46:20,414 UTC [INFO ] ([ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)') - com.emc.documentum.rest.util.LogHelper : XMLOutputFactory loaded com.ctc.wstx.stax.WstxOutputFactory.
2021-04-26 06:46:20,416 UTC [INFO ] ([ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)') - com.emc.documentum.rest.util.LogHelper : XMLInputFactory loaded com.ctc.wstx.stax.WstxInputFactory.
2021-04-26 06:46:20,451 UTC [INFO ] ([ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)') - com.emc.documentum.rest.util.LogHelper : Class com.emc.documentum.rest.config.DataBindingRuntime addLastPropertySource rest-api-data-binding.properties.
The recently deployed release contained many things but looking into it in details, the most promising suspect was the Oracle WebLogic Server PSU (+coherence patch) from April 2021. Based on the logs, this looked like a GUAVA (Google core libraries for Java) related issue. Usually, the D2-REST application would be using its own application libraries but it might happen that for some security reasons, the configuration would be changed to force WebLogic to use the Oracle provided ones instead. This would be in order to keep the third-party libraries up-to-date, as much as possible, to reduce the potential security issues. At this customer, rollback the PSU would be a rather important security problem. After looking into the details, it was clear that the method mentioned above has been deleted in GUAVA 21.0 (deprecated in 20.0). On the other hand, D2 16.5.1 comes with GUAVA 13.0.1 by default and D2-REST (+ D2-Smartview) comes with GUAVA 20.0. As part of the April PSU, this library was probably upgraded to 21.0 (I didn’t find any confirmation). Therefore, I tried to force D2-REST to re-use its internal GUAVA libraries instead (while keeping the others from WebLogic) by adding a new line inside the “<prefer-application-packages>” section:
[weblogic@wsd2rest-0 ~]$ cd $APPLICATIONS/D2-REST/WEB-INF/
[weblogic@wsd2rest-0 WEB-INF]$ cat weblogic.xml
<?xml version="1.0" encoding="UTF-8"?>
<weblogic-web-app>
...
<container-descriptor>
<!--prefer-web-inf-classes>true</prefer-web-inf-classes-->
<prefer-application-packages>
<package-name>org.slf4j</package-name>
<package-name>com.google.common.*</package-name>
</prefer-application-packages>
<!--show-archived-real-path-enabled>true</show-archived-real-path-enabled-->
</container-descriptor>
...
</weblogic-web-app>
[weblogic@wsd2rest-0 WEB-INF]$
Adding the line 11 above forces WebLogic to load the application specific packages instead of its own. After a Managed Server restart, the issue was gone, which confirms that the April PSU was the culprit:
Since we force WebLogic to not use its own jar files for some Google libraries, that means that potential security issues related to these jar files are obviously re-opened… However, at some point, you have a choice to make between being secure but having a non-working application OR potentially having some flaws but a working application. It’s obviously possible to go one-step further and instead of using “<package-name>com.google.common.*</package-name>“, which is rather generic, use a more refined definition of the package so that the scope affected is smaller.
The same applies to D2-Smartview as well since it is also a REST client, so it relies heavily on such packages…
![Thumbnail [60x60]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/MOP_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/OLS_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/GME_web-min-scaled.jpg)