Top 10 WebLogic Performance Metrics to Proactively Monitor a Server Farm

My 10 Key Metrics
Now – let me get into some of the key areas I personally monitor and an explanation of why I monitor them. Note, they are not in any specific order. The list below is only a partial list to provide an example of the type of data you can use to tune, troubleshoot and learn about how your system performs.
#1: JVM – Percent of time in Garbage Collection
Time Spent in GC is a key indicator of the apps way to use memory
Time Spent in GC is a key indicator of the apps way to use memory
GC is a stop the world process. So it is very important to verify the system is not spending too much time in this state. This metric is also helpful in validating configuration changes and for capacity management. Factors that go into this include system cores, memory allocated etc…

<a target="_blank" href="https://www.amazon.in/b?_encoding=UTF8&tag=malli12-21&linkCode=ur2&linkId=91df7e8327372224af36ea81f59c0b29&camp=3638&creative=24630&node=1375248031">https://weblogicadmintutorials.blogspot.in/</a><img src="//ir-in.amazon-adsystem.com/e/ir?t=malli12-21&l=ur2&o=31" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" />
#2: Execute Thread Counts
Watch the number of threads as they give an indication on how well your system runs
Watch the number of threads as they give an indication on how well your system runs
When WebLogic opens up too many threads to service the load, there is a decrease in performance. Threads take resources (CPU, Memory). This metric can be used for monitoring and also for capacity planning.
#3: Workmanager Thread usage
Workmanager Threads ensure resources are properly assigned to applications
Workmanager Threads ensure resources are properly assigned to applications
A Workmanager is used to limit resources or to ensure that the right application gets the resources. Here is where you can validate the Workmanagers are not capped at an inappropriate level etc…
#4: JDBC
Determine whether you have proper sizing and isolate connection leaks in your app
Determine whether you have proper sizing and isolate connection leaks in your app
Current Capacity vs. Current Capacity High allows you to validate that you have the correct amount of resources available to service the client’s needs. It’s also helpful to determine if you need to increase or decrease the pool size. While Connection Delay Times can be used to determine DB responsiveness.
#5: Application Health and Applications deployed
Actively monitor your application health and not just rely on the WebLogic Health Status
Actively monitor your application health and not just rely on the WebLogic Health Status
Validate all deployments are deployed to the correct servers and are in an active state. I can’t count the number of times Weblogic said everything was Active but the server it was deployed to said “Failed.” IT can also start monitoring active sessions to the servers. This is great data for capacity planning.
#6a: JMS Oldest Message Age
Old messages in a queue means that your system can't keep up with processing them
Old messages in a queue means that your system can’t keep up with processing them
Normally it doesn’t matter how many messages are on the queue, it is a good idea to pay close attention to how old the oldest message is. This is normally a key indication of issues, or shows that the system is being affected by an excessive message dump on the queue – something the system cannot keep up with the load (capacity). This can be verified (below).
#6b: JMS Consumers
Make sure to monitor all key metrics for JMS
Make sure to monitor all key metrics for JMS
In the picture above, you are able to see how many consumers are on the queue. If no consumers are visible, then we don’t process messages.
#7: Cluster Server Alive Count
This metric ensures all servers in the cluster are talking and know about each other.
#8: Server Listen Address
Verify availability and responsiveness of the server listening port.
Verify availability and responsiveness of the server listening port.
This metric allows you to know if all servers are communicating properly to the Admin Server. If the listen address is not reporting properly (<host>/<IP>) the managed server is not communicating with the Admin Server. You lose the ability to monitor and also troubleshoot through the console. Normally this happens when the server is under extremely heavy load or, depending on your Weblogic version, it is a Weblogic bug.
#9: Server Running Time
Catch servers that keep restarting due to crashes by watching the total run time
Catch servers that keep restarting due to crashes by watching the total run time
This is a great metric to catch servers that crash and get restarted by Nodemanager. :-)
#10: Monitoring Time
Keep an eye on the overhead of your own monitoring
Keep an eye on the overhead of your own monitoring
You want to limit monitoring so that you leave the maximum resources available for the system. I’m often asked how to figure out the right balance of system monitoring. Every system is different so there is no magic number. A few key things to look at:
§  Are your servers properly sized? Is there enough CPU/Memory available?
§  You have monitoring in place, on the individual servers, so you can validate whether or not you are placing undue load on those servers.
§  With the Dynatrace agents you can compare response times to your monitoring intervals to see if you are straining the system or notice any performance impact.
Those pieces of information can help you determine the appropriate amount of monitoring. In a future blog, maybe we can cover these in more detail.


WebLogic Performance Metrics to Proactively Monitor a Server Farm

In my years of experience with Weblogic monitoring, I came up with a list of key metrics that can help determine the health of my server farm. These early indicators allow me to set pro-active steps instead of waiting for end users to complain.
The following screenshot shows one of my Dynatrace dashboards containing key health metrics captured through JMX:
In my years of experience with Weblogic monitoring, I came up with a list of key metrics that can help determine the health of my server farm. These early indicators allow me to set pro-active steps instead of waiting for end users to complain.
The following screenshot shows one of my Dynatrace dashboards containing key health metrics captured through JMX:
Dynatrace Dashboard showing the health status of all main components in the Weblogic domain: Deployment State, Server Health, Threads, JVM, JDBC, JMS etc…
Dynatrace Dashboard showing the health status of all main components in the Weblogic domain: Deployment State, Server Health, Threads, JVM, JDBC, JMS etc…
Metrics to become Proactive instead of Reactive
Monitoring is a proactive, not reactive, approach to system management. My philosophy is that IT should know about system issues before any customer ever calls to notify us. In fact, IT needs to be notifying internal and external customers using our systems that they may experience a decrease in performance prior to customers noticing the system degradation. This allows a stronger working relationship between IT and users/customers. It also decreases the amount of firefighting needed and reduces the number of people needed to isolate and resolve system issues. While you must initially take the time to learn your system and gather data, it is an invaluable investment of time and pays huge dividends in terms of system efficiency.
The following is another dashboard I use to analyze a specific metric area. In this case it tells me my system is impacted by hogging/stuck threads on 2 of my 6 JVMs in the cluster:
Detailed information about Thread States makes it easy to identify problems such as stuck threads or even Workmanager capacity issues. This allows us to take proactive steps prior to customer complaints.
Detailed information about Thread States makes it easy to identify problems such as stuck threads or even Workmanager capacity issues. This allows us to take proactive steps prior to customer complaints.

WebLogic 12c: Node Manager Best Practices

During the last couple of years (and the last couple of WebLogic versions) I collected a number of best practices  regarding WebLogic nodemanager. All of them hold true for WebLogic 12c as well. This posting is not a step-by-step beginners guide and it will not save you from attending some training or studying the Oracle documentation regarding node manager yourself. Anyway, here are some suggestions, check if the apply for your environments:

Node Manager Best Practices


  • At first, take a decision to start servers with or without NM. Note, that  is not absolutely necessary. You can always start your servers with the scripts generated by the config wizzard. I personally know rather big companies building lovely cars who took the decsicion not to use node manager.
  • Would I use nodemanger myself? For an “average” project: yes! Only after configuring node manager you can use the WebLogic admin console to start and stop managed servers and node manager will restart you failed servers as well. However, if you consider restarting you servers automatically because of out-of-memory problems, better read this article about “surviving generations”  to understand how to track down memory leaks and fix them. Anyway, you still want to use node manager.
    https://weblogicadmintutorials.blogspot.in/
  • Make sure you understand that nodemanger will use default values to start your servers unless you specify them yourself in the admin console under server startup parameters.
  • Make sure you always start your servers with same startup parameters! This is really important. You end up in deep trouble if you don’t. Believe me.
    Imagine somebody is starting a managed server using the admin console and the provided values there. Next day somebody else starts a server using the provided scripts (which – at least in real life – will never be identical to the startup values configured in the admin console). Now depending on the way the server was started it will behave differently and show erratic behaviour or not.
  • Document and communicate the usage of node manager. Write it down in the operations manual. If you ever hire me as a consultant for some performance tuning it helps to know if you are actually using node manager or not.
  • Don’t forget to enroll new machines for NM usingnmEnroll()
  • A good way to overcome the potential problem with  startup parameters configured in admin console is to use:
    ?
    1
    2
    [file: nodemanger.properties]startScriptEnabled=true
    stopScriptEnabled=true
    Then node manager will use the generated start script and you do not need to configure startup values in the admin server console.
  • If you are not using SSL for your domain the default option for node manager to use encrypted communication does not make that much sense for you. Disable it. On the admin server site switcht to “plain text” for node manager communication and in the node manager.properties located in WL_HOME/common/nodemanager set
    ?
    1
    secureListener=false
  • If you decide to use SSL for the node manager communication, get correct certificates! The demo certificate will not work in a distributed system. Make sure the hostnames in the certificates are correct. If they are not correct, you may want to consider disabling host name verification on the admin server (which is the client for the node manager).
  • Remember that node manager is not part of the domain. Still you can check the node manager status and and see the logs directly from the WebLogic admin console.
    https://weblogicadmintutorials.blogspot.in/

How to migrate projects from Ant to Maven tool

For some time we were thinking about migrating our build to maven from ant. It happened last month and was actually simpler than what we have anticipated. From my experience, here is a brief about the steps we have followed. Our application is a  is a enterprise web application build with multiple frameworks and technologies and is deployed as a single WAR.

1. Create maven project directory structure.
As told in the Maven user guide create the below directory structure. We have done it under a new folder for the project.

https://weblogicadmintutorials.blogspot.in/


2. Move the files/folders keeping the SCM logs.

Even though the folder structure is new the source files will be old ones! We want to keep the SCM logs while moving them to new locations. Remember to commit the folders created in step 1 before you start moving your files. If you use SVN, see this user guide or SO question on how to do this. Move the java source, unit/integration test and configuration resources to appropriate folders.

3. Create the POM and add dependencies

Most critical part in the migration is adding the dependencies in the POM. Start by adding the dependencies for the frameworks used in your application. Make sure you are adding the right version of the jars. You can find the version of the jar by reading the MANIFEST.MF file inside the META-INF folder of the jar. This will help if the versions are missing from the file name.

Any third party jars can be added to the maven repository as told here. If you are using very old versions of jar files some of them may not be available in maven repository.Here you can either try upgrading to newer versions or prepare local install as told before. Once you have added all the dependencies try building the application. Watch out for any major issues.
4. Make sure you haven't changed much in the WAR

Maven is a build tool. This means your WAR should not change. So, in the last step we compare both versions and make sure they are the same. Make sure you are on top of all the differences. Also, compare the jar files generated by maven and your existing files, Sync them by
     - Adding <exclusions> to remove the unwanted jars
     - Add the dependencies for the missing jars
This can be a tiring tasks depending on the number of jars you have in your lib. But make sure that you have each one covered and know that why they exists in your app.
May  be this is a late post, most applications might have already been migrated by now. Anyways, better late than never! According to many experts Gradle is also a good choice as a build tool for your new project.



- See more at: http://blog.manupk.com/2014/02/4-simple-steps-to-migrate-legacy.html#sthash.pTR5mG6p.dpuf