It was a long time that I wrote my last blog. Lots of customer activities and I had no time to write one. With the acquired knowledges, it’s time to write more blogs and share knowledge, don’t you think ?

Let’s begin with an easy one. During customer activity when upgrading a complex Fusion Middleware Platform, it was asked to us to provide support during a move to a secure zone. I can’t tell you what’s really behind the secure zone, probably a more protected and more restricted network with more firewall restriction, don’t ask I have no idea but we unfortunately had an issue. Before that move, I was quite confident as it was informed that it’s only impacting the current IP address. No stress as all the Fusion Middleware Component configuration were using a network alias leveraging the impact of any network changes that could happen. So as a sample for this blog post, the WebLogic instances listen addresses was set with the network alias “dbi-cust-1983.dbi-services.com” pointing to the real hostname in the DNS “vmtestdbiofm01.dbi-services.com”. The NodeManager was set the same for the configured machine. Please see some screenshot to help to understand the configuration we had.

WebLogic Admin Server Listen Address

ofm-wls-firewall-AdminServer-ListenAddress

NodeManager machine Listen Address

ofm-wls-firewall-NM-ListenAddress

NodeManager Listen Address

weblogic@:/home/weblogic/ [dbiOFMHDV] cd $DOMAIN_HOME
weblogic@:/data/weblogic/config/domains/dbiOFMHDV/ [dbiOFMHDV] find ./ -name nodemanager.properties
./nodemanager/nodemanager.properties
weblogic@:/data/weblogic/config/domains/dbiOFMHDV/ [dbiOFMHDV] cat nodemanager/nodemanager.properties | grep ListenAddress
ListenAddress=dbi-cust-1983.dbi-services.com

Let’s also add that the network naming resolution for the Middleware Components was also done through the DNS and the local naming resolution (/etc/hosts) wasn’t containing the real host nor the DNS alias used.

weblogic@:/data/weblogic/config/domains/dbiOFMHDV/ [dbiOFMHDV] cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

So the operation began, we shutdown all the Middleware and Database Components, the Network and Unix Team worked on their tasks (IP changes, DNS correlated changes, forcing changes propagation over the network, aso). Once they completed and once we confirmed that Network Config change have been properly applied (ifconfig, ping, nslookup), we started the platform again without any changes from the Middleware part. Application team performed smoke testing and there was no functional impact.

After reviewing the logs for any error, I found an Error with the RJVM module as follow

####<Apr 16, 2018, 11:08:06,438 AM CEST> <Error> <RJVM> <vmtestdbiofm01.dbi-services.com> <AdminServer> <ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <d41ced14-a5e8-4ef9-bd90-19f63910849d-00000059> <1523869686438> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000572> <The server rejected a connection attempt JVMMessage from: '3739718027889105070S:dbi-cust-1983.dbi-services.com:[-1,-1,9001,9001,-1,-1,-1]:dbiOFMHDV:WLS_FORMS' to: '0B:10.32.0.12:[8443,-1,-1,-1,-1,-1,-1]' cmd: 'CMD_IDENTIFY_REQUEST', QOS: '101', responseId: '-1', invokableId: '-1', flags: 'JVMIDs Sent, TX Context Not Sent, 0x1', abbrev offset: '183' probably due to an incorrect firewall configuration or administrative command.>
####<Apr 16, 2018, 11:08:55,371 AM CEST> <Error> <RJVM> <vmtestdbiofm01.dbi-services.com> <AdminServer> <ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <d41ced14-a5e8-4ef9-bd90-19f63910849d-0000005d> <1523869735371> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000572> <The server rejected a connection attempt JVMMessage from: '-2190410908851642855S:dbi-cust-1983.dbi-services.com:[-1,-1,9002,9002,-1,-1,-1]:dbiOFMHDV:WLS_REPORTS' to: '0B:10.32.0.12:[8443,-1,-1,-1,-1,-1,-1]' cmd: 'CMD_IDENTIFY_REQUEST', QOS: '101', responseId: '-1', invokableId: '-1', flags: 'JVMIDs Sent, TX Context Not Sent, 0x1', abbrev offset: '183' probably due to an incorrect firewall configuration or administrative command.>
####<Apr 16, 2018, 11:09:06,509 AM CEST> <Error> <RJVM> <vmtestdbiofm01.dbi-services.com> <AdminServer> <ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <d41ced14-a5e8-4ef9-bd90-19f63910849d-0000005e> <1523869746509> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000572> <The server rejected a connection attempt JVMMessage from: '3739718027889105070S:dbi-cust-1983.dbi-services.com:[-1,-1,9001,9001,-1,-1,-1]:dbiOFMHDV:WLS_FORMS' to: '0B:10.32.0.12:[8443,-1,-1,-1,-1,-1,-1]' cmd: 'CMD_IDENTIFY_REQUEST', QOS: '101', responseId: '-1', invokableId: '-1', flags: 'JVMIDs Sent, TX Context Not Sent, 0x1', abbrev offset: '183' probably due to an incorrect firewall configuration or administrative command.>
####<Apr 16, 2018, 11:09:59,162 AM CEST> <Error> <RJVM> <vmtestdbiofm01.dbi-services.com> <AdminServer> <ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <d41ced14-a5e8-4ef9-bd90-19f63910849d-00000059> <1523869799162> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000572> <The server rejected a connection attempt JVMMessage from: '-2190410908851642855S:dbi-cust-1983.dbi-services.com:[-1,-1,9002,9002,-1,-1,-1]:dbiOFMHDV:WLS_REPORTS' to: '0B:10.32.0.12:[8443,-1,-1,-1,-1,-1,-1]' cmd: 'CMD_IDENTIFY_REQUEST', QOS: '101', responseId: '-1', invokableId: '-1', flags: 'JVMIDs Sent, TX Context Not Sent, 0x1', abbrev offset: '183' probably due to an incorrect firewall configuration or administrative command.>

Facing this issue, I ask colleagues for any recommendation or finding to solve the issue ASAP. They recommended to have a look on the cluster configuration for any broadcasting messaging mode, the default cluster configuration messaging mode of my Middleware component was unicast.

WebLogic Server – Default Forms & Reports Cluster configuration

ofm-wls-firewall-Cluster-ListUnicast

Let’s check the BEA error code to continue the investigation.

The BEA-00572 error description

ofm-wls-firewall-BEA-ErrorDescription

The message is quite clear “The server rejected a connection attempt JVMMessage From …” As said before, all the Fusion Middleware was properly started and the functional tests shown that there was no impact at all. It’s always recommended to find a solution to remove such kind of error message. After some research on the MOS, I found a document “Doc ID 860340.1” applicable the WebLogic Server since the version 7.0. It clearly described that this issue was not reported, nor tested on more recent WebLogic Server version 10.3 or 12c version. I was not really confident with this Oracle Support Note but I tested in case of and I was surprised.

The provided solution by Oracle work very well on my case, I append the  JAVA Option with the flag “-Dweblogic.rjvm.enableprotocolswitch=true” in the setUserOverrides.sh script located under the $DOMAIN_HOME/bin.

export JAVA_OPTIONS="${JAVA_OPTIONS} -Ddomain.home=/u02/weblogic/config/domains/dbiOFMHDV -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.security.SSL.minimumProtocolVersion=TLSv1.2 -Dweblogic.security.disableNullCipher=true -Djava.security.egd=file:///dev/./urandom -Dweblogic.rjvm.enableprotocolswitch=true"

After this change and after having restarted the whole domain, there error was gone. No more probable Firewall Configuration Error.

Hope this will help other people that will have to move a Fusion Middleware 12c Platform into a secure zone.