Sometimes you have java processes or even jboss servers using a lot of CPU. In my example I had an xPlore dsearch server using like 98% of the cpu. When using jconsole and jvisualvm I figured out the garbage collector was using 50 to 60% of the cpu time.
This was because the server was indexing and accessing the internal DB very often. Hence a lot of objects were created, the JVM was not correctly sized, thus all objects went to the tenured(old) space resulting in filling up the heap. The garbage collector had to go through the whole heap and perform a lot of FUll GC. I went to a point where I had a Full GC every 5 seconds that lasted 4 seconds. So I had only 1 second every 5 seconds of “real” processing.

So if you have a process stuck in collecting garbage you can use the following parameters:
USER_MEM_ARGS=”-Xms8G -Xmx8G -XX:PermSize=64m -XX:NewSize=2g -XX:MaxNewSize=2g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -XX:+CMSClassUnloadingEnabled -XX:CMSInitiatingOccupancyFraction=95 -XX:+UseCMSInitiatingOccupancyOnly -XX:MaxTenuringThreshold=2 -XX:MaxPermSize=256m -Xss1024k -Xloggc:/pkgs/dmsp/opt/documentum/xPlore/jboss7.1.1/server/DctmServer_PrimaryDsearch/logs/PrimaryJVM.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution”

-Xms8G: Starting limit of the size of JVM
-Xmx8G: Max limit of heap usage
-XX:PermSize=64m: Permanent space size
-XX:MaxPermSize=256m:
-XX:NewSize=2g: Young Gen space size, here 1/4 of total
-XX:MaxNewSize=2: Maximum Young Gen space
-XX:+UseParNewGC: Parallele copying collector for Young generation, parallelizes the collection process with multiple threads, better perf with multiple processor architecture
-XX:+UseConcMarkSweepGC: Use concurrent mark-sweep collection for the old generation
-XX:+CMSParallelRemarkEnabled: Parallelize the remark phase, goes with CMS option, increases response time
-XX:+ParallelRefProcEnabled: Parallelize the process of weak referenced objects (cache)
-XX:+CMSClassUnloadingEnabled: Enables the class unloading capability for CMS
-XX:CMSInitiatingOccupancyFraction=95: Puts the limit after which the full GC will be trigered, higher value means less Full GC but longuer
-XX:+UseCMSInitiatingOccupancyOnly: Prevents the JVM tu use heuristics GC triggering rules, sets the trigger to use only the previous percentage as a threeshold for Full GC trigger.
-XX:MaxTenuringThreshold=2: Maximum value for tenuring threshold. The default value is 15
-Xss1024k: Thread stack size

These arguments will help you see the behaviour of the GC process:
-Xloggc:/path/to/log/JVM.log
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintTenuringDistribution

 

After tuning the JVM I went back to a normal behaviour:

jvm-all-green