WebLogic Server along with the JVM offers various ways to collect logs, debug traces, diagnostic images, thread dumps, and much more. However, useful data can easily get buried with lots of noise, unneeded data that won't help resolving a particular problem.
Performance Issues
We will start with performance issues such as long running requests or server hangs. The best way to narrow down where the issue lies is to dump the java threads and parse them through a thread dump analyzer tool such as ThreadLogic. Looking at the server log files, or enabling some debugging features would be premature since the first step is to identify the thread(s) that are stuck to determine what is being executed and relevant JEE resources. The log could also alert about stuck threads once they have been stuck for over the maximum allowed time (see MaxStruckThreadTime) but unless thread dumps are examined, it won't be possible to determine possible patterns of issues and threads' relationship around locked objects.
Memory Leaks
Another common performance issue involves memory leaks. If you encounter OutOfMemory java.lang exceptions you should issue a heap dump and analyze it with a tool such as VisualVM or the Eclipse MAT. See my post on Heap Dumps analysis for more details. Heap dumps are snapshots of memory certain times. Heap dumps are to the Java process memory (non native) what thread dumps are to Java process threads. Enabling -verbose:gc is also an effective way to monitor heap usage at runtime. The output of the GC stats can be redirected to a file so that not to clog the server logs. This can be done by setting -Xloggc with HotSpot JVM. It's recommended to include -XX:+PrintGCDetails and -XX:+PrintGCTimeStamps to log time stamps and detailed GC activities as well. Troubleshooting guide for HotSpot VM details these flags among others useful flags for diagnosis purposes.
Applications, 3rd parties, and configuration issues
The following chart summarizes what could be collected to debug each of the listed WLS component. This list is by no means exhaustive. Many more debug flags could be used for these listed components. This chart contains examples of useful debug options and links to additional resources.
For the following, Redirect stdout and stderr logging should be enabled. This can be done via the Administration Console in the advanced section of the servers logging tab or at startup in the java command line with -Dweblogic.StdoutDebugEnabled=true. In addition, the severity level of the log file should be set to debug.
WLS comp. or 3rd party resource
|
Examples of Debug options and flags
|
MyOracleSupport related resources
(login required)
|
EJB/Web Serv.
|
-Dweblogic.webservice.verbose=true
-Dweblogic.webservice.client.verbose=true
To log SOAP requests and response messages
Trace in servers log files
| |
JMS
|
-Dweblogic.debug.DebugJMSStore
To log info on load, store events and transaction records into servers log files
See Debugging JMS for more JMS debugging options
| |
SSL
|
-Dssl.debug=true
-Dweblogic.security.SSL.verbose=true
Trace in servers log files
| |
Clustering
|
-Dweblogic.debug.DebugReplication for high level replication info
-Dweblogic.debug.DebugClusterAnnouncements to log info on announcement, StateDump and attributes messages sent or received by multicast
Trace in servers log files
| |
JTA
|
weblogic.JTAXA & weblogic.JTA2PC for runtime issues, example:
JAVA_OPTIONS="-Dweblogic.Debug=weblogic.JTAXA, weblogic.JTA2PC"
Trace in servers log files
| |
JDBC
|
-Dweblogic.debug.DebugJDBCSQL=true
to print information about JDBC methods invoked with their arguments, return values, and thrown exceptions
See JDBC Debugging Scopes under Monitoring JDBC Resources
Trace in servers log files
| |
Oracle JDBC Driver (*)
|
-Doracle.jdbc.Trace=true to enable jdbc logging
Create OracleLog.properties
Trace is file defined in oracle.jdbc.LogFile
| |
Deployment (Classloader)
|
Set the following options to true to report which classes are getting loaded
-Dweblogic.utils.classloaders.GenericClassLoader.Verbose
-Dweblogic.utils.classloaders.ChangeAwareClassLoader.Verbose
-Dweblogic.utils.classloaders.ClasspathClassFinder
-Dweblogic.utils.classloaders.DefaultFilteringClassLoader.Verbose
-Dweblogic.utils.classloaders.FilteringClassLoader.Verbose
-Dweblogic.utils.classloaders.FilteringClassLoader.ResourceDump
Trace in servers log files
| |
Proxy plug-in
|
Debug="ALL" in the proxy configuration file
(iisproxy.ini, httpd.conf or obj.conf)
Trace in file set in WLLogFile defined inside proxy configuration file
|
(*) Enabling JDBC Driver debug flags is verbose so it's recommended to set logging properties to appropriate levels such as FINE, SEVERE, FINEST, INFO etc. based upon the debugging requirement.
WLDF
The Weblogic Diagnostic Framework can be used to collect metrics, setup watch and notifications, and to define instrumentation. With WLS 12.1.2, you can set the level (Low, Medium or High) of built-in modules that will gather metrics, or disable them completely. WLDF collects runtime statistics on JDBC, JTA and JVM and more. You can then use the Monitoring Dashboard to view and navigate through the collected data. WLDF is not an alternative to debug options. Debug options activate tracing that is coded as part of WLS whereas WLDF collects and monitors runtime mbeans values and can activate notifications with defined rules. They are used for different purposes.
The WLDF built-in configurations (Low, Medium, & High) can be cloned into a new configuration, and used as templates for creating custom WLDF configurations. The Low volume built-in is enabled by default in production mode. One new feature with WLS 12.1.2 is Runtime control which gives the ability to activate or deactivate diagnostic system modules dynamically at runtime wihtout making any domain configuration change.
JFR
JFR recording data can also be captured through WLDF diagnostic images based on WLDF watch rules so that JVM runtime system information can be analyzed along with recording of WLS components diagnostic data. Once extracted from the diagnostic image capture, the JFR file can be analyzed using Java Mission Control.
WLST
WLST diagnostic commands can be used to extract data from the diagnostic archive (event and harvested metric data) in either XML form or CSV, based on the function you use. You can also dump the diagnostics data from a harvester to a local file.
The following WLST functions allow to capture an image and to download image files from a remote system over WLST:
One major difference between the diagnostic framework and the debug options is that debugs are enabled so that an issue can be reproduced and traced whereas the diagnostic image capture is used as a server-level dump for post-failure analysis in a similar way heap dumps are used post memory leak failures.
jstack can be use to dump the java threads. While working with WebLogic Server, you can dump the java threads with various tools such as WLST. jstack can do just that but differs from <ctrl>+<break> or kill -3 since it doesn't dump a heap summary along with the threads.
In addition, jstack can print lock information with -l option and the list of synchronizers that may be exclusively owned by a thread. Such information can be very useful while monitoring concurrent access by threads.
jstack comes as part of the JDK installed with WebLogic Server. With instances of WLS running, you can open a new shell, execute the setDomain script and run jstack or any tool that's part of the JDK package. Thread dumps taken by jstack can be redirected to files, e.g. jstack <pid> > myTD.txt.
PIDs can be obtained on Solaris/Linux with the ps (report process status) command, e.g. ps -ef | grep java, and from the task managed on Windows. In task manager you might need to go to View and Select Columns to add PID or process identifiers.
Thread dumps
Introduction
A Java thread dump is a snapshot what every thread in the JVM is doing at a particular point in time. Each thread in the JVM is listed with it's name and id, it's current state and the Java call stack showing what monitor it has locked or is waiting on. This is especially useful if your Java application sometimes seems to hang when running under load, as an analysis of the dump will show where the threads are stuck, either by deadlock or other thread contention.
How to get a thread dump
Unix and Linux
On most flavors of Unix kill -3 <pid> will generate a thread dump. To get the java process pid use ps -ef | grep java. The dump is directed to stdout so redirecting to a file is neccessary. Since 1.6 the Sun/Oracle JDK has the jstack utility to do the same thing.
Enter the following at the command line at startup:bin\jrcmd.exe <pid> print_threads
Windows:
If the application was started from a command window you can select the window and use CTRL+Breakbut the output will likely overflow the window. Jstack is available if your using the Sun JDK. Otherwise use PsExec.exe ( http://technet.microsoft.com/en-us/sysinternals/bb897553.aspx ) to execute SendSignal.exe.
How many thread dumps do I need
Often a single thread dump will reveal the problem, this is especially true with deadlocks where two or more threads are waiting for locks obtained by each other. But in other cases, such as threads waiting on other processes, like IO from a database, you won't be able to detect it with a single thread dump. In this case multiple thread dumps taken over time will show the same thread waiting for the process to complete during that time span.
Understanding Thread States
Before starting on dissecting thread dumps it is very important to get a good understanding of the common thread states.
- NEW : A thread is in a new state when it has just been created and the start() method hasn't been invoked
- RUNNING : A thread is in a runnable state when the start method has been invoked and the JVM actively schedules it for execution
- BLOCKED / WAITING FOR MONITOR ENTRY: Thread enters this state when it isn't able to acquire the necessary monitors to enter a synchronized block. A thread waits for other threads to reliquish the monitor. A thread in this state isnt doing any work
- WAITING : A thread enters this state when it relinquishes its control over the monitor by calling Object.wait(). It waits till the time other threads invoke Object.notify() or Object.notifyAll(). In this state too the thread isnt doing any work
- TIMED WAITING : A thread enters this state when it relinquishes its control over the monitor by calling Object.wait(long). It waits till the time other threads invoke Object.notify() / Object.notifyAll() or when the wait time expires. In this state too the thread isnt doing any work
- SLEEPING / WAITING ON CONDITION: A thread sleeps when it calls Thread.sleep(long). As the name suggests the thread isnt doing in this state either
- Terminated
- Other
As you can see from the above list a thread is only doing active work when it is in the RUNABBLE state. In all other states the thread is just wasting CPU cycles. For maximum throughput from an application it is necessary to ensure that the maximum number of threads are in the RUNNABLE state and fewer threads in WAITING FOR MONITOR ENTRY, SLEEPING, WAITING or TIMED WAITING state.
l Threads present in a typical weblogic thread dump
l JVM threads { GC threads, Code optimisation etc }
l ListenThread.Default
l Weblogic.socket.Muxer
l Weblogic.kernel.System
l Weblogic.admin.RMI
l Weblogic.admin.HTTP
l Weblogic.kernel.Defaul
l VDE Transaction Processor Thread
l Weblogic.health.CoreHealthMonitor
l Miscellaneous Threads ( Timer, Security etc. )
Analyzing the thread dumps
You can read a thread dump with a text editor, and many people use only this.
The Beginning of The Thread Dump
The thread dump starts with the date and time of the dump, and the version number of the JRockit JVM used (see Listing 20-1).
===== FULL THREAD DUMP =============== Wed Feb 21 13:46:45 2007 BEA JRockit(R) R27.1.0-109-73164-1.5.0_08-20061129-1428-windows-ia32
Format of thread dumps
Full thread dump Java HotSpot(TM) Client VM (1.5.0_04-b05 mixed mode, sharing):
"Thread-1" prio=5 tid=0x00a995d0 nid=0x1300 in Object.wait()[0x02d0f000..0x02d0fb68] at java.lang.Object.wait(Native Method) - waiting on <0x22aaa8d0> (a org.tw.testyard.thread.Drop) at java.lang.Object.wait(Unknown Source) at org.tw.testyard.thread.Drop.take(Drop.java:14) - locked <0x22aaa8d0> (a org.tw.testyard.thread.Drop) at org.tw.testyard.thread.Consumer.run(Consumer.java:15) at java.lang.Thread.run(Unknown Source) "Thread-0" prio=5 tid=0x00a88440 nid=0x6a4 waiting on condition[0x02ccf000..0x02ccfbe8] at java.lang.Thread.sleep(Native Method) at org.tw.testyard.thread.Producer.run(Producer.java:24) at java.lang.Thread.run(Unknown Source) |
In the thread dump snippet above you can observe the following
- The thread dump starts with "Full thread dump", followed by a list of threads currently being executed.
- There are 2 application threads called Thread-1 and Thread-0. These are the default names which the JVM handed over to the threads.
- "Thread-1" is waiting for a notification after it called Object.wait() on the Drop object.
- Similarly, "Thread-0" is sleeping on a condition after it called Thread.sleep
- At this particular instant, there are no threads in the runnable state and hence the application isn't doing any real work.
Although thread dumps also lists the state of the system threads we are not going to look deeper into system threads.
Samurai description
A GUI based tail, thread dump analysis tool
Samurai is a tabbed GUI tail application. Samurai will colors idle threads in gray, blocked threads in red and all the running threads in green.
There are three result views and Samurai shows "Table view" by default. In many case you are just interested in the table view and the sequence view.
Use the table view to decide which thread needs be inspected, the sequence view to understand the thread's behavior. You should take care especially threds always in red.
Here are some key features of "Samurai":
Picks thread dumps from std/stderr output and colorize them.
Visualize the "-verbose:gc" log.
Works as a GUI based "tail -f"
Requirements:
JDK 1.4 or later
_______________________________________________________________________________
Download: https://java.net/projects/threadlogic/downloads
Thread Dump Analysis is a key tool for performance tuning and troubleshooting
of Java based applications. The current set of TDA tools (Samurai/TDA) dont
mine the thread dumps or provide a more detailed view of what each thread is
doing while just limiting themselves to reporting the state (locked/waiting/running)
or the lock information.
Once a thread dump is parsed and threads details are populated, each of the
thread is then analyzed against matching advisories and tagged appropriately.
The threads are also associated with specific Thread Groups based on
functionality or thread group name. Both the advisories and grouping are
managed via xml definition files which can be modified or extended.
Each of the advisory has a health level indicating severity of the issue found,
pattern, name, keyword and related advice.
Thread Groups
One can see the thread groups are divided into two buckets - WLS and non-WLS
related threads. The JVM threads, LDAP and other unknown custom threads go
under the non-WLS bucket while all the WLS, Muxer, ADF, Coherence, Oracle,
SOA, JMS, Oracle Adapter threads are all under the WLS bucket. The
classification can be changed by modifying the GroupsDefn xml files.
The health levels (in descending of severity) are FATAL (meant for Deadlocks,
STUCK, Finalizer blocked etc), WARNING, WATCH (worth watching), NORMAL
and IGNORE. Based on the highest severity of threads within a group, that
health level gets promoted to the Thread Group's health level and same is
repeated at the thread dump level. There can be multiple advisories tagged to a
Thread, Thread Group and Thread Dump.
Each of the advisory gets triggered based on either call execution patterns
observed in the thread stack or presence of other conditions (thread blocked or
multiple threads blocked for same lock can trigger BlockedThreads Advisory).
Sometimes a thread might be tagged as IGNORE or NORMAL based on its
execution logic or might be tagged more specifically as involved in JMS send or
receive client or a Servlet thread. The advisories are generated based on an
advisory xml map that is extensible.
Heap dump:
A Heap Dump is a snapshot of the Java process heap memory at a given time. It's a useful tool to troubleshoot memory leaks and other memory related issues. Heap dumps are to memory what thread dumps are to java threads.
There are different ways to dump the java heap used by a Java application such as WebLogic Server. Heap dumps can be triggered automatically by JVMs such as HotSpot or JRockit when an OutOfMemory event occurs.
Example:
java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:\myApp\hprof-dumps myApp
This command will generate a heap dump file in the defined heap dump directory when an OOM is thrown.
The file will be in HPROF binary format (*.hprof). If no path is defined the file will be created in the directory from which the java process was launched. The format of the file with JRockit is jrockit_pid,hprof, pid being the process ID of this JVM instance. With HotSpot the format of the file will be java_pidpid.hprof.
Example:
java.lang.OutOfMemoryError: Java heap space
Dumping heap to D:\java_pid4148.hprof ...
Heap dump file created [24921731 bytes in 1.400 secs]
For WebLogic Server these properties, HeapDumpOnOutOfMemoryError and HeapDumpPath(optional), can be added directly to the startup script that contain the Java command that launches WLS. On Windows the script (with default name startWebLogic.cmd) lives in the domain bin directory and the parameters can be directly added to the existing JAVA_OPTIONS.
With a JRockit VM process the following command can also be used to generate a heap dump.
Example:
jrcmd pid hprofdump filename=name_of_dump_file
However if you are using this command against a non JRockit process the following error will come up:
java.io.IOException: Command failed in target VM
Example:
jmap -heap:live,format=b,file=filename pid
This command will generate a heap dump file in binary format and will only count live objects (the live option is optional). Live objects are used by applications and generally reside in the Young Generation part of the heap.
As illustrated in the screenshot below, generating a heap dump with VisualVM is very simple. VisualVM is shipped as part of the HotSpot JVM (/bin/jvisualvm.exe). However, VisualVM can connect to java processes that started with other JVM such as JRockit.
Once generated, the heap dump will be parsed as shown below:
VisualVM enables you to browse heap dumps as shown above with the Classes view where you can easily identify the number and percentage of instances referred by each class.
jhat can also be used to analyze heap dump files.
jhat can also be used to analyze heap dump files.
Example:
jhat dumpfile
Reading from dumpfile...
Dump file created Fri May 17 15:25:41 PDT 2013
Snapshot read, resolving...
Resolving 580390 objects...
Chasing references, expect 116 dots.............................................
.......................................................................
Eliminating duplicate references................................................
....................................................................
Snapshot resolved.
Started HTTP server on port 7000
Server is ready.
Dump file created Fri May 17 15:25:41 PDT 2013
Snapshot read, resolving...
Resolving 580390 objects...
Chasing references, expect 116 dots.............................................
.......................................................................
Eliminating duplicate references................................................
....................................................................
Snapshot resolved.
Started HTTP server on port 7000
Server is ready.
Then you can open http://localhost:7000/ to navigate through the content of the dump file to identify objects with high number of occurrences.
Finally, as shown below, the OpenSource Eclipse MAT offers a very rich GUI that includes histograms and leak hunter actions, overview and leak suspect reports and much more.
__________________________________________________________________________________________
jhat - Java Heap Analysis Tool
SYNOPSIS
PARAMETERS
DESCRIPTION
OPTIONS
jhat - Java Heap Analysis Tool
· Synopsis
· Options
· See Also
SYNOPSIS
jhat [ options ] <heap-dump-file>
PARAMETERS
Options, if used, should follow immediately after the command name.
heap-dump-file
Java binary heap dump file to be browsed. For a dump file that contains multiple heap dumps, you may specify which dump in the file by appending "#<number> to the file name, i.e. "foo.hprof#3".
DESCRIPTION
The jhat command parses a java heap dump file and launches a webserver. jhat enables you to browse heap dumps using your favorite webbrowser. jhat supports pre-designed queries (such as 'show all instances of a known class "Foo"') as well as OQL (Object Query Language) - a SQL-like query language to query heap dumps. Help on OQL is available from the OQL help page shown by jhat. With the default port, OQL help is available at http://localhost:7000/oqlhelp/
There are several ways to generate a java heap dump:
· Use jmap -dump option to obtain a heap dump at runtime;
· Use jconsole option to obtain a heap dump via HotSpotDiagnosticMXBean at runtime;
· Heap dump will be generated when OutOfMemoryError is thrown by specifying -XX:+HeapDumpOnOutOfMemoryError VM option;
· Use hprof.
NOTE: This tool is experimental and may not be available in future versions of the JDK.
OPTIONS
-stack false/true
Turn off tracking object allocation call stack. Note that if allocation site information is not available in the heap dump, you have to set this flag to false. Default is true.
-refs false/true
Turn off tracking of references to objects. Default is true. By default, back pointers (objects pointing to a given object a.k.a referrers or in-coming references) are calculated for all objects in the heap.
-port port-number
Set the port for the jhat's HTTP server. Default is 7000.
-exclude exclude-file
Specify a file that lists data members that should be excluded from the "reachable objects" query. For example, if the file lists java.lang.String.value, then, whenever list of objects reachable from a specific object "o" are calculated, reference paths involving java.lang.String.value field will not considered.
-baseline baseline-dump-file
Specify a baseline heap dump. Objects in both heap dumps with the same object ID will be marked as not being "new". Other objects will be marked as "new". This is useful while comparing two different heap dumps.
-debug int
Set debug level for this tool. 0 means no debug output. Set higher values for more verbose modes.
-version
Report version number and exit.
-h
Output help message and exit.
-help
Output help message and exit.
-J<flag>
Pass <flag> to the Java virtual machine on which jhat is run. For example, -J-Xmx512m to use a maximum heap size of 512MB.
CORE DUMP
Core dump is usually a Binary file which gets generrated by the operating system when JVM or any other
process crashes. some times it also happens that the jvm will not be able to generate the crash dump.
Path:
Windows: c:\Documents and settings\All Users\Application Data\Microsoft\Dr Watson.
By default in Unix based operating systems the core dump files are created in the directory where the Java
program/server was started even sometimes it is generated in the "/Tmp" directory of the operating system.
but we can change the location to create crash/Heap dump.
-XX:HeapDumpPath=/opt/app/Prefered Location and -XX:+Heap Dump jvm Options.
Filename:
Sund JDK: "hs_err_pid<weblogicPID>.log
for JRockit Jvm also generates a textual file with name "*.dump"
Flags:
Sun JDK JAVA_OPTIONS -XX:+showMessageBoxOnError.
JRockit JAVA_OPTIONS -Djrockit.waitonerror.
Cause JVM Crash.
1.Native Code causes the JVM crash (c/C++,JNI-Javanative interface APIs)
2.JDBC Drivers Specially the Native drivers
3.JVM code optimization
4.Less Memory availability for native area of a java process.
5.Application Server Native Performance Pack Libraries.
6.JVMs library itself can cause the Crash.
7.High CPU Utilization by the threads.
Tools.
1. window's : Dr.Watson start->run->drwtsn32.
2. Solaris : Pstack and Pmap
3. Linux : lsstack