Documentum – Unable to restart a JMS, failed deployment of bundle.jar

While working on a very big Documentum project with several other teams, some people were complaining that the JMS wouldn’t start anymore on one of the DEV environments. It is kind of rare to face an issue with the JMS itself (JBoss works pretty well usually…) so I was interested in checking this. This was a Documentum 7.2 environment but I’m sure it would apply to older versions as well as newer versions (7.x, 16.4, …). This was the content of the JMS console output:

[dmadmin@content_server_01 ~]$ cat $JMS_HOME/server/nohup-JMS.out
...
2018-06-12 07:15:54,765 UTC INFO  [org.jboss.modules] JBoss Modules version 1.1.1.GA
2018-06-12 07:15:54,968 UTC INFO  [org.jboss.msc] JBoss MSC version 1.0.2.GA
2018-06-12 07:15:55,019 UTC INFO  [org.jboss.as] JBAS015899: JBoss AS 7.1.1.Final "Brontes" starting
2018-06-12 07:15:56,415 UTC INFO  [org.jboss.as.server] JBAS015888: Creating http management service using socket-binding (management-http)
2018-06-12 07:15:56,417 UTC INFO  [org.xnio] XNIO Version 3.0.3.GA
2018-06-12 07:15:56,428 UTC INFO  [org.xnio.nio] XNIO NIO Implementation Version 3.0.3.GA
2018-06-12 07:15:56,437 UTC INFO  [org.jboss.remoting] JBoss Remoting version 3.2.3.GA
2018-06-12 07:15:56,448 UTC INFO  [org.jboss.as.logging] JBAS011502: Removing bootstrap log handlers
2018-06-12 07:15:56,478 UTC INFO  [org.jboss.as.configadmin] (ServerService Thread Pool -- 27) JBAS016200: Activating ConfigAdmin Subsystem
2018-06-12 07:15:56,486 UTC INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 32) JBAS010280: Activating Infinispan subsystem.
2018-06-12 07:15:56,510 UTC INFO  [org.jboss.as.osgi] (ServerService Thread Pool -- 40) JBAS011940: Activating OSGi Subsystem
2018-06-12 07:15:56,518 UTC INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 39) JBAS011800: Activating Naming Subsystem
2018-06-12 07:15:56,527 UTC INFO  [org.jboss.as.security] (ServerService Thread Pool -- 45) JBAS013101: Activating Security Subsystem
2018-06-12 07:15:56,533 UTC INFO  [org.jboss.as.security] (MSC service thread 1-9) JBAS013100: Current PicketBox version=4.0.7.Final
2018-06-12 07:15:56,535 UTC INFO  [org.jboss.as.connector] (MSC service thread 1-4) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.9.Final)
2018-06-12 07:15:56,548 UTC INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 49) JBAS015537: Activating WebServices Extension
2018-06-12 07:15:56,585 UTC INFO  [org.jboss.as.naming] (MSC service thread 1-1) JBAS011802: Starting Naming Service
2018-06-12 07:15:56,594 UTC INFO  [org.jboss.as.mail.extension] (MSC service thread 1-12) JBAS015400: Bound mail session 
2018-06-12 07:15:56,688 UTC INFO  [org.jboss.ws.common.management.AbstractServerConfig] (MSC service thread 1-8) JBoss Web Services - Stack CXF Server 4.0.2.GA
2018-06-12 07:15:56,727 UTC INFO  [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-10) Starting Coyote HTTP/1.1 on http--0.0.0.0-9080
2018-06-12 07:15:56,821 UTC INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-13) live server is starting with configuration HornetQ Configuration (clustered=false,backup=false,sharedStore=true,journalDirectory=$JMS_HOME/server/DctmServer_MethodServer/data/messagingjournal,bindingsDirectory=$JMS_HOME/server/DctmServer_MethodServer/data/messagingbindings,largeMessagesDirectory=$JMS_HOME/server/DctmServer_MethodServer/data/messaginglargemessages,pagingDirectory=$JMS_HOME/server/DctmServer_MethodServer/data/messagingpaging)
2018-06-12 07:15:56,827 UTC INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-13) Waiting to obtain live lock
2018-06-12 07:15:56,852 UTC INFO  [org.hornetq.core.persistence.impl.journal.JournalStorageManager] (MSC service thread 1-13) Using AIO Journal
2018-06-12 07:15:56,923 UTC INFO  [org.hornetq.core.server.impl.AIOFileLockNodeManager] (MSC service thread 1-13) Waiting to obtain live lock
2018-06-12 07:15:56,924 UTC INFO  [org.hornetq.core.server.impl.AIOFileLockNodeManager] (MSC service thread 1-13) Live Server Obtained live lock
2018-06-12 07:15:56,981 UTC INFO  [org.jboss.as.remoting] (MSC service thread 1-5) JBAS017100: Listening on 0.0.0.0/0.0.0.0:9092
2018-06-12 07:15:56,982 UTC INFO  [org.jboss.as.remoting] (MSC service thread 1-14) JBAS017100: Listening on /127.0.0.1:9084
2018-06-12 07:15:56,996 UTC INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-10) JBAS015012: Started FileSystemDeploymentService for directory $JMS_HOME/server/DctmServer_MethodServer/deployments
2018-06-12 07:15:57,019 UTC INFO  [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015014: Re-attempting failed deployment bundle.jar
2018-06-12 07:15:57,020 UTC INFO  [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015014: Re-attempting failed deployment bundle.jar
2018-06-12 07:15:57,021 UTC INFO  [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015014: Re-attempting failed deployment bundle.jar
2018-06-12 07:15:57,023 UTC INFO  [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015014: Re-attempting failed deployment bundle.jar
2018-06-12 07:15:57,024 UTC INFO  [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015014: Re-attempting failed deployment bundle.jar
2018-06-12 07:15:57,025 UTC INFO  [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015014: Re-attempting failed deployment bundle.jar
2018-06-12 07:15:57,026 UTC INFO  [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015003: Found ServerApps.ear in deployment directory. To trigger deployment create a file called ServerApps.ear.dodeploy
2018-06-12 07:15:57,027 UTC INFO  [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015003: Found XhiveConnector.ear in deployment directory. To trigger deployment create a file called XhiveConnector.ear.dodeploy
2018-06-12 07:15:57,028 UTC INFO  [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015003: Found bpm.ear in deployment directory. To trigger deployment create a file called bpm.ear.dodeploy
2018-06-12 07:15:57,029 UTC INFO  [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015003: Found acs.ear in deployment directory. To trigger deployment create a file called acs.ear.dodeploy
2018-06-12 07:15:57,090 UTC INFO  [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-2) Starting Coyote HTTP/1.1 on http-0.0.0.0-0.0.0.0-9082
2018-06-12 07:15:57,235 UTC INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-13) Started Netty Acceptor version 3.2.5.Final-a96d88c 0.0.0.0:9090 for CORE protocol
2018-06-12 07:15:57,237 UTC INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-13) Started Netty Acceptor version 3.2.5.Final-a96d88c 0.0.0.0:9091 for CORE protocol
2018-06-12 07:15:57,239 UTC INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-13) Server is now live
2018-06-12 07:15:57,239 UTC INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-13) HornetQ Server version 2.2.13.Final (HQ_2_2_13_FINAL_AS7, 122) [b774a781-a4da-11e6-a1e8-005056082847]) started
2018-06-12 07:15:57,256 UTC INFO  [org.jboss.as.messaging] (MSC service thread 1-14) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/RemoteConnectionFactory
2018-06-12 07:15:57,257 UTC INFO  [org.jboss.as.messaging] (MSC service thread 1-14) JBAS011601: Bound messaging object to jndi name java:/RemoteConnectionFactory
2018-06-12 07:15:57,258 UTC INFO  [org.jboss.as.messaging] (MSC service thread 1-4) JBAS011601: Bound messaging object to jndi name java:/ConnectionFactory
2018-06-12 07:15:57,261 UTC INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-9) trying to deploy queue jms.queue.touchRpcQueue
2018-06-12 07:15:57,267 UTC INFO  [org.jboss.as.messaging] (MSC service thread 1-9) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/queue/touchRpcQueue
2018-06-12 07:15:57,268 UTC INFO  [org.jboss.as.messaging] (MSC service thread 1-9) JBAS011601: Bound messaging object to jndi name java:/queue/touchRpcQueue
2018-06-12 07:15:57,269 UTC INFO  [org.jboss.as.messaging] (MSC service thread 1-3) JBAS011601: Bound messaging object to jndi name java:/TouchRpcQueueConnectionFactory
2018-06-12 07:15:57,270 UTC INFO  [org.jboss.as.messaging] (MSC service thread 1-3) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/TouchRpcQueueConnectionFactory
2018-06-12 07:15:57,301 UTC INFO  [org.jboss.as.deployment.connector] (MSC service thread 1-8) JBAS010406: Registered connection factory java:/JmsXA
2018-06-12 07:15:57,311 UTC INFO  [org.hornetq.ra.HornetQResourceAdapter] (MSC service thread 1-8) HornetQ resource adaptor started
2018-06-12 07:15:57,312 UTC INFO  [org.jboss.as.connector.services.ResourceAdapterActivatorService$ResourceAdapterActivator] (MSC service thread 1-8) IJ020002: Deployed: file://RaActivatorhornetq-ra
2018-06-12 07:15:57,316 UTC INFO  [org.jboss.as.deployment.connector] (MSC service thread 1-4) JBAS010401: Bound JCA ConnectionFactory 
2018-06-12 07:15:57,343 UTC ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 2) Operation ("add") failed - address: ([("deployment" => "bundle.jar")]) - failure description: "JBAS014803: Duplicate resource [("deployment" => "bundle.jar")]"
2018-06-12 07:15:57,348 UTC ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS014654: Composite operation was rolled back
2018-06-12 07:15:57,349 UTC INFO  [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:9085
2018-06-12 07:15:57,350 UTC ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS014654: Composite operation was rolled back
2018-06-12 07:15:57,350 UTC INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS 7.1.1.Final "Brontes" started in 2885ms - Started 148 of 226 services (78 services are passive or on-demand)
2018-06-12 07:15:57,351 UTC ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS014654: Composite operation was rolled back
2018-06-12 07:15:57,353 UTC ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS014654: Composite operation was rolled back
2018-06-12 07:15:57,354 UTC ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS014654: Composite operation was rolled back
2018-06-12 07:15:57,355 UTC ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) {"JBAS014653: Composite operation failed and was rolled back. Steps that failed:" => {"Operation step-1" => "JBAS014803: Duplicate resource [("deployment" => "bundle.jar")]"}}
2018-06-12 07:15:57,357 UTC ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) undefined
2018-06-12 07:15:57,358 UTC ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) undefined
2018-06-12 07:15:57,359 UTC ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) undefined
2018-06-12 07:15:57,359 UTC ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) undefined

This was linked to deployments so I tried as a first step to force a redeploy of all Documentum applications. It can be done as shown below and in those cases, it is better to stop the JMS before otherwise it will try to redeploy the applications on the fly, which might work but you aren’t sure they will start again on next startup… So here are the steps:

[dmadmin@content_server_01 ~]$ cd $JMS_HOME/server/DctmServer_MethodServer/deployments
[dmadmin@content_server_01 deployments]$ 
[dmadmin@content_server_01 deployments]$ find . -maxdepth 1 -name "*.failed"
./acs.ear.failed
./ServerApps.ear.failed
./error.war.failed
./bpm.ear.failed
[dmadmin@content_server_01 deployments]$ 
[dmadmin@content_server_01 deployments]$ 
[dmadmin@content_server_01 deployments]$ for i in `find . -maxdepth 1 -name "*.failed"`; do name=`echo ${i} | sed 's,.failed,.dodeploy,'`; mv ${i} ${name}; done
[dmadmin@content_server_01 deployments]$

Then you can just start again the JMS:

[dmadmin@content_server_01 deployments]$ $JMS_HOME/server/startJMS.sh
Starting the JMS...
The JMS process has been started.
[dmadmin@content_server_01 deployments]$

Unfortunately in this case, the JMS logs were similar and the Documentum applications were still in failed. So looking closer at these logs, it seemed to be related to a bundle.jar file:

[dmadmin@content_server_01 deployments]$ find . -name bundle.jar
./felix-cache/bundle3/version0.0/bundle.jar
./felix-cache/bundle5/version0.0/bundle.jar
./felix-cache/bundle1/version0.0/bundle.jar
./felix-cache/bundle4/version0.0/bundle.jar
./felix-cache/bundle6/version0.0/bundle.jar
./felix-cache/bundle2/version0.0/bundle.jar
[dmadmin@content_server_01 deployments]$
[dmadmin@content_server_01 deployments]$ ls felix-cache/*
felix-cache/bundle0:
bundle.id

felix-cache/bundle1:
bundle.id  bundle.lastmodified  bundle.location  bundle.startlevel  bundle.state  version0.0

felix-cache/bundle2:
bundle.id  bundle.lastmodified  bundle.location  bundle.startlevel  bundle.state  version0.0

felix-cache/bundle3:
bundle.id  bundle.lastmodified  bundle.location  bundle.startlevel  bundle.state  version0.0

felix-cache/bundle4:
bundle.id  bundle.lastmodified  bundle.location  bundle.startlevel  bundle.state  version0.0

felix-cache/bundle5:
bundle.id  bundle.lastmodified  bundle.location  bundle.startlevel  bundle.state  version0.0

felix-cache/bundle6:
bundle.id  bundle.lastmodified  bundle.location  bundle.startlevel  bundle.state  version0.0
[dmadmin@content_server_01 deployments]$
[dmadmin@content_server_01 deployments]$
[dmadmin@content_server_01 deployments]$ for i in `ls felix-cache/`; do echo "felix-cache/${i}:"; cat felix-cache/${i}/bundle.location; echo; echo; done
felix-cache/bundle0:
cat: felix-cache/bundle0/bundle.location: No such file or directory


felix-cache/bundle1:
file:$JMS_HOME/server/DctmServer_MethodServer/deployments/acs.ear/lib/HttpService.jar

felix-cache/bundle2:
file:$JMS_HOME/server/DctmServer_MethodServer/deployments/acs.ear/lib/HttpServiceImpl.jar

felix-cache/bundle3:
file:$JMS_HOME/server/DctmServer_MethodServer/deployments/acs.ear/lib/Common.jar

felix-cache/bundle4:
file:$JMS_HOME/server/DctmServer_MethodServer/deployments/acs.ear/lib/Web.jar

felix-cache/bundle5:
file:$JMS_HOME/server/DctmServer_MethodServer/deployments/acs.ear/lib/Jmx.jar

felix-cache/bundle6:
file:$JMS_HOME/server/DctmServer_MethodServer/deployments/acs.ear/lib/org.apache.felix.bundlerepository-1.2.1.jar

[dmadmin@content_server_01 deployments]$
[dmadmin@content_server_01 deployments]$ diff felix-cache/bundle1/version0.0/bundle.jar $JMS_HOME/server/DctmServer_MethodServer/deployments/acs.ear/lib/HttpService.jar
[dmadmin@content_server_01 deployments]$
[dmadmin@content_server_01 deployments]$ diff felix-cache/bundle3/version0.0/bundle.jar $JMS_HOME/server/DctmServer_MethodServer/deployments/acs.ear/lib/Common.jar
[dmadmin@content_server_01 deployments]$

This felix-cache is normally created by the ACS when you start it and it will normally be placed in the folder from where you start the JMS. In this case, it was under the deployments folder, probably because someone started the JMS using the startMethodServer.sh script directly and he didn’t use our custom start script (startJMS.sh as you can see above and below) which actually takes care of that and switch to the correct folder before (and with the proper nohup, aso…). Because it was created under the deployments folder, the JMS was trying to deploy it as any other applications and so it failed. As you can see above, the file “bundle.jar” (in the different folders) is actually a copy of some of the ACS library files – the ones mentioned in the bundle.location – that has been put to the felix-cache by the ACS.

As you know, with a newly installed JMS for Documentum, there is no felix-cache folder under the deployments. What you can find here are the ServerApps, ACS or BPM for example but that’s pretty much it. Therefore, to solve this issue, simply remove the felix-cache folder, force a new deployment of all Documentum applications, restart it again properly and it should be good then:

[dmadmin@content_server_01 deployments]$ $JMS_HOME/server/stopMethodServer.sh; sleep 5; ps -ef | grep MethodServer
[dmadmin@content_server_01 deployments]$
[dmadmin@content_server_01 deployments]$ rm -rf felix-cache/
[dmadmin@content_server_01 deployments]$
[dmadmin@content_server_01 deployments]$ find . -maxdepth 1 -name "*.failed"
./acs.ear.failed
./ServerApps.ear.failed
./error.war.failed
./bpm.ear.failed
[dmadmin@content_server_01 deployments]$
[dmadmin@content_server_01 deployments]$ for i in `find . -maxdepth 1 -name "*.failed"`; do name=`echo ${i} | sed 's,.failed,.dodeploy,'`; mv ${i} ${name}; done
[dmadmin@content_server_01 deployments]$
[dmadmin@content_server_01 deployments]$ $JMS_HOME/server/startJMS.sh
Starting the JMS...
The JMS process has been started.
[dmadmin@content_server_01 deployments]$

Then checking if the issue is gone and if all the applications are now properly deployed:

[dmadmin@content_server_01 deployments]$ grep -E " deployment | Deployed " $JMS_HOME/server/nohup-JMS.out
2018-06-12 07:30:15,142 UTC INFO  [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015003: Found ServerApps.ear in deployment directory. To trigger deployment create a file called ServerApps.ear.dodeploy
2018-06-12 07:30:15,144 UTC INFO  [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015003: Found XhiveConnector.ear in deployment directory. To trigger deployment create a file called XhiveConnector.ear.dodeploy
2018-06-12 07:30:15,145 UTC INFO  [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015003: Found bpm.ear in deployment directory. To trigger deployment create a file called bpm.ear.dodeploy
2018-06-12 07:30:15,146 UTC INFO  [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015003: Found acs.ear in deployment directory. To trigger deployment create a file called acs.ear.dodeploy
2018-06-12 07:30:15,503 UTC INFO  [org.jboss.as.connector.services.ResourceAdapterActivatorService$ResourceAdapterActivator] (MSC service thread 1-1) IJ020002: Deployed: file://RaActivatorhornetq-ra
2018-06-12 07:30:15,535 UTC INFO  [org.jboss.as.server.deployment] (MSC service thread 1-5) JBAS015876: Starting deployment of "bpm.ear"
2018-06-12 07:30:15,535 UTC INFO  [org.jboss.as.server.deployment] (MSC service thread 1-13) JBAS015876: Starting deployment of "ServerApps.ear"
2018-06-12 07:30:15,535 UTC INFO  [org.jboss.as.server.deployment] (MSC service thread 1-7) JBAS015876: Starting deployment of "error.war"
2018-06-12 07:30:15,535 UTC INFO  [org.jboss.as.server.deployment] (MSC service thread 1-11) JBAS015876: Starting deployment of "acs.ear"
2018-06-12 07:30:17,824 UTC INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015876: Starting deployment of "documentum-bocs-ws.war"
2018-06-12 07:30:17,825 UTC INFO  [org.jboss.as.server.deployment] (MSC service thread 1-15) JBAS015876: Starting deployment of "bocs.war"
2018-06-12 07:30:18,093 UTC INFO  [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/ACS]] (MSC service thread 1-16) Initializing CORS filter as per the deployment descriptor
2018-06-12 07:30:18,608 UTC INFO  [org.jboss.as.server.deployment] (MSC service thread 1-11) JBAS015876: Starting deployment of "DmMethods.war"
2018-06-12 07:30:18,608 UTC INFO  [org.jboss.as.server.deployment] (MSC service thread 1-9) JBAS015876: Starting deployment of "DmMail.war"
2018-06-12 07:30:18,904 UTC INFO  [org.jboss.as.server.deployment] (MSC service thread 1-11) JBAS015876: Starting deployment of "bpm.war"
2018-06-12 07:30:32,017 UTC INFO  [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS018559: Deployed "bpm.ear"
2018-06-12 07:30:32,019 UTC INFO  [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS018559: Deployed "error.war"
2018-06-12 07:30:32,020 UTC INFO  [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS018559: Deployed "acs.ear"
2018-06-12 07:30:32,021 UTC INFO  [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS018559: Deployed "ServerApps.ear"
[dmadmin@content_server_01 deployments]$

So takes care when you start the JMS, if you do not have any custom script (it’s pretty much mandatory for the JMS!) or if you do not change the working directory before executing the startMethodServer.sh script, you might have some surprises.

Edit: After writing this blog, I searched the OTX website for something related to this and found that there is a KB (KB7795388) about this but I think this blog still makes sense because it provides more information and some explanation. That’s a bad habit of mine, I usually try fixing things myself before checking the OTX website because I don’t like it and the less I’m using it, the better.