WebLogic Performance Metrics to Proactively Monitor a Server Farm

In my years of experience with Weblogic monitoring, I came up with a list of key metrics that can help determine the health of my server farm. These early indicators allow me to set pro-active steps instead of waiting for end users to complain.
The following screenshot shows one of my Dynatrace dashboards containing key health metrics captured through JMX:
In my years of experience with Weblogic monitoring, I came up with a list of key metrics that can help determine the health of my server farm. These early indicators allow me to set pro-active steps instead of waiting for end users to complain.
The following screenshot shows one of my Dynatrace dashboards containing key health metrics captured through JMX:
Dynatrace Dashboard showing the health status of all main components in the Weblogic domain: Deployment State, Server Health, Threads, JVM, JDBC, JMS etc…
Dynatrace Dashboard showing the health status of all main components in the Weblogic domain: Deployment State, Server Health, Threads, JVM, JDBC, JMS etc…
Metrics to become Proactive instead of Reactive
Monitoring is a proactive, not reactive, approach to system management. My philosophy is that IT should know about system issues before any customer ever calls to notify us. In fact, IT needs to be notifying internal and external customers using our systems that they may experience a decrease in performance prior to customers noticing the system degradation. This allows a stronger working relationship between IT and users/customers. It also decreases the amount of firefighting needed and reduces the number of people needed to isolate and resolve system issues. While you must initially take the time to learn your system and gather data, it is an invaluable investment of time and pays huge dividends in terms of system efficiency.
The following is another dashboard I use to analyze a specific metric area. In this case it tells me my system is impacted by hogging/stuck threads on 2 of my 6 JVMs in the cluster:
Detailed information about Thread States makes it easy to identify problems such as stuck threads or even Workmanager capacity issues. This allows us to take proactive steps prior to customer complaints.
Detailed information about Thread States makes it easy to identify problems such as stuck threads or even Workmanager capacity issues. This allows us to take proactive steps prior to customer complaints.

WebLogic 12c: Node Manager Best Practices

During the last couple of years (and the last couple of WebLogic versions) I collected a number of best practices  regarding WebLogic nodemanager. All of them hold true for WebLogic 12c as well. This posting is not a step-by-step beginners guide and it will not save you from attending some training or studying the Oracle documentation regarding node manager yourself. Anyway, here are some suggestions, check if the apply for your environments:
https://amzn.to/421ko7G

Node Manager Best Practices


  • At first, take a decision to start servers with or without NM. Note, that  is not absolutely necessary. You can always start your servers with the scripts generated by the config wizzard. I personally know rather big companies building lovely cars who took the decsicion not to use node manager.
  • Would I use nodemanger myself? For an “average” project: yes! Only after configuring node manager you can use the WebLogic admin console to start and stop managed servers and node manager will restart you failed servers as well. However, if you consider restarting you servers automatically because of out-of-memory problems, better read this article about “surviving generations”  to understand how to track down memory leaks and fix them. Anyway, you still want to use node manager.
    https://weblogicadmintutorials.blogspot.in/
  • Make sure you understand that nodemanger will use default values to start your servers unless you specify them yourself in the admin console under server startup parameters.
  • Make sure you always start your servers with same startup parameters! This is really important. You end up in deep trouble if you don’t. Believe me.
    Imagine somebody is starting a managed server using the admin console and the provided values there. Next day somebody else starts a server using the provided scripts (which – at least in real life – will never be identical to the startup values configured in the admin console). Now depending on the way the server was started it will behave differently and show erratic behaviour or not.
  • Document and communicate the usage of node manager. Write it down in the operations manual. If you ever hire me as a consultant for some performance tuning it helps to know if you are actually using node manager or not.
  • Don’t forget to enroll new machines for NM usingnmEnroll()
  • A good way to overcome the potential problem with  startup parameters configured in admin console is to use:
    ?
    1
    2
    [file: nodemanger.properties]startScriptEnabled=true
    stopScriptEnabled=true
    Then node manager will use the generated start script and you do not need to configure startup values in the admin server console.
  • If you are not using SSL for your domain the default option for node manager to use encrypted communication does not make that much sense for you. Disable it. On the admin server site switcht to “plain text” for node manager communication and in the node manager.properties located in WL_HOME/common/nodemanager set
    ?
    1
    secureListener=false
  • If you decide to use SSL for the node manager communication, get correct certificates! The demo certificate will not work in a distributed system. Make sure the hostnames in the certificates are correct. If they are not correct, you may want to consider disabling host name verification on the admin server (which is the client for the node manager).
  • Remember that node manager is not part of the domain. Still you can check the node manager status and and see the logs directly from the WebLogic admin console.
    https://weblogicadmintutorials.blogspot.in/

How to migrate projects from Ant to Maven tool

For some time we were thinking about migrating our build to maven from ant. It happened last month and was actually simpler than what we have anticipated. From my experience, here is a brief about the steps we have followed. Our application is a  is a enterprise web application build with multiple frameworks and technologies and is deployed as a single WAR.

1. Create maven project directory structure.
As told in the Maven user guide create the below directory structure. We have done it under a new folder for the project.

https://weblogicadmintutorials.blogspot.in/


2. Move the files/folders keeping the SCM logs.

Even though the folder structure is new the source files will be old ones! We want to keep the SCM logs while moving them to new locations. Remember to commit the folders created in step 1 before you start moving your files. If you use SVN, see this user guide or SO question on how to do this. Move the java source, unit/integration test and configuration resources to appropriate folders.

3. Create the POM and add dependencies

Most critical part in the migration is adding the dependencies in the POM. Start by adding the dependencies for the frameworks used in your application. Make sure you are adding the right version of the jars. You can find the version of the jar by reading the MANIFEST.MF file inside the META-INF folder of the jar. This will help if the versions are missing from the file name.

Any third party jars can be added to the maven repository as told here. If you are using very old versions of jar files some of them may not be available in maven repository.Here you can either try upgrading to newer versions or prepare local install as told before. Once you have added all the dependencies try building the application. Watch out for any major issues.
4. Make sure you haven't changed much in the WAR

Maven is a build tool. This means your WAR should not change. So, in the last step we compare both versions and make sure they are the same. Make sure you are on top of all the differences. Also, compare the jar files generated by maven and your existing files, Sync them by
     - Adding <exclusions> to remove the unwanted jars
     - Add the dependencies for the missing jars
This can be a tiring tasks depending on the number of jars you have in your lib. But make sure that you have each one covered and know that why they exists in your app.
May  be this is a late post, most applications might have already been migrated by now. Anyways, better late than never! According to many experts Gradle is also a good choice as a build tool for your new project.



- See more at: http://blog.manupk.com/2014/02/4-simple-steps-to-migrate-legacy.html#sthash.pTR5mG6p.dpuf

JMS

An enterprise messaging system enables applications to communicate with one another through the exchange of messages. A message is a request, report, and/or event that contains information needed to coordinate communication between different applications. A message provides a level of abstraction, allowing you to separate the details about the destination system from the application code.
The Java Message Service (JMS) is a standard API for accessing enterprise messaging systems. Specifically, JMS


WebLogic Server Domains

WebLogic Server Domains

Best deals on luggage and backpacks The basic administrative unit for a WebLogic Server installation is called a domain. A domain is a logically related group of WebLogic Server resources that you manage as a unit. A domain always includes only one instance of WebLogic Server called the Administration Server. The Administration Server serves as a central point of contact for server instances and system administration tools. A domain may also include additional WebLogic Server instances called Managed Servers.
You can configure some or all of these Managed Servers to be part of a WebLogic Server cluster. A cluster is a group of WebLogic Server instances that work together to provide scalability and high-availability for applications. A Managed Server in a cluster can act as a backup for services such as JMS and JTA that are hosted on another server instance in the cluster. Your applications are also deployed and managed as part of a domain.
A Managed Server can also function as a virtual host.
You can organize your domains based on criteria such as:
  • Logical divisions of applications. For example, a domain devoted to end-user functions such as shopping carts and another domain devoted to back-end accounting applications.
  • Physical location. Domains for different locations or branches of your business.
  • Size. Domains organized in small units that can be managed more efficiently, perhaps by different personnel.
For more information about WebLogic Server domains, see:
    WebLogic Server Domain
  • Overview of WebLogic Server Domains
  • Creating and Configuring Domains Using the Configuration Wizard
  • Configuring Domains in the Administration Console 
  • Figure 1-1 WebLogic Server Domain

     
Figure 1-1 depicts a possible configuration of a WebLogic Server Domain—one of many possible configurations.
In the depicted domain, there are three physical machines:
Machine A hosts one instance of WebLogic Server, the Administration Server. The System Administration Tools communicate with the Administration Server to perform configuration and monitoring of the servers and applications in the domain. The Administration Server communicates with each Managed Server on behalf of the System Administration Tools. The configuration for all the servers in the domain is stored in the configuration repository, the config.xml file, which resides on the machine hosting the Administration Server.
Machines B and C each host two instances of WebLogic Server, WebLogic Servers 1 through 4. These instances are called Managed Servers. The Administration Server communicates with an instance of Node Manager running on each machine to control startup and shutdown of the Managed Servers.
WebLogic Servers 2 and 4 are part of a WebLogic Cluster (outlined in red). This cluster is running an application that responds to HTTP requests routed to the cluster from a hardware load balancer. (An instance of WebLogic Server or a third-party Web server with one of the WebLogic Server plug-ins can also provide load balancing.) The load balancer processes HTTP requests from the Internet after they have passed through a firewall. The load balancer and firewall are not part of the domain. A replicated copy of objects such as HTTP sessions is passed between the two cluster members to provide failover capability.
WebLogic Server 1 runs an application that uses Java Database Connectivity (JDBC) to access a database server running on another physical machine that is not part of the WebLogic Domain.
Note: The pictured domain is only intended to illustrate the concepts of a WebLogic Server domain and how you manage the domain. Many possible configurations of servers, clusters, and applications are possible in a WebLogic Server domain.


System Administration Infrastructure

System administration infrastructure in WebLogic Server is implemented using the Java Management Extension (JMX) specification from Sun Microsystems. The JMX API models system administration functions with Java objects called MBeans. Knowledge of this implementation as described in this discussion of system administration infrastructure is not necessary to manage a WebLogic Server domain.
There are three types of MBeans used to manage a WebLogic Server domain: administrationconfiguration, and runtime Mbeans.
Administration Mbeans contain a set of attributes that define configuration parameters for various management functions. All attributes for administration MBeans have pre-set default values. When the Administration Server starts, it reads the domain configuration file (called config.xml) and overrides the default attribute values of the administration MBeans with any attribute values found in the config.xml file.
The config.xml file, located on the machine that hosts the Administration Server, provides persistent storage of Mbean attribute values. Every time you change an attribute using the system administration tools, its value is stored in the appropriate administration MBean and written to the config.xml file. Each WebLogic Server domain has its own config.xml file.
Each time the Administration Server starts successfully, and each time you modify the configuration, a backup configuration file is created. You can configure the number of back up copies ofconfig.xml retained by the Administration Server. For more information, see Backing up config.xml.
If you set any configuration attributes on the command line when you start the Administration Server with -D arguments, these values override the values set by the defaults or those read from the config.xml file. These overridden values are also persisted to config.xml file by the Administration Server. For more information about these command-line arguments, see Configuring Servers in the Administration Console Online Help.
Configuration Mbeans are copies of Administration Mbeans that each Managed Server uses to initialize its configuration. When you start a Managed Server, the server receives a copy of all the administration MBeans configured on the Administration Server and stores them in memory as configuration MBeans. If you override any configuration attributes when starting a Managed Server, those values override the values received from the Administration Server but are not written to the config.xml file. For more information about starting a Managed Server, see Starting Managed Servers in the Administration Console Online Help.
Runtime Mbeans contain sets of attributes consisting of runtime information for active WebLogic Servers instances and applications. By retrieving the values of attributes in these runtime MBeans, you can monitor the running status of a WebLogic Server domain.
Mbeans may also contain operations used to execute management functions.
Although users with a knowledge of these Mbeans and the JMX API can create their own customized management system, most users prefer to use the system administration tools provided with WebLogic Server to perform these tasks. These tools do not require knowledge of the JMX API. For more information, see System Administration Tools.

Introduction to System Administration

You manage a WebLogic Server installation by using any of several system administration tools provided with WebLogic Server. A WebLogic Server installation can consist of a single WebLogic Server instance or multiple instances, each hosted on one or more physical machines. The system administration tools include the Administration Console, command line utilities, and an API, with which you manage security, database connections, messaging, transaction processing, and the runtime configuration of your applications. The tools also allow you to monitor the health of the WebLogic Server environment to ensure maximum availability and performance for your applications.

Finding latest Patchset for OFMW SOA components !!!

Finding latest Patchset for OFMW SOA components !!!

Oracle recommend to patch SOA environments with latest patch set whenever it’s available. Its SOA administrator responsibility to apply latest patch set in all SOA environments time to time in order to make sure Oracle product remain to latest patch set levels. 

In each patch set there is list of bug which Oracle fixes, often I have seen if we raise a service ticket in Oracle for any specific bug, they will ask straight away what version of product component getting used.

They also ask to provide inventory logs for each Oracle product components e.g. SCA, OSB, WLS and Oracle_COMMON. 

In below blog I will show very simple example how to check current product level version of SOA bundle and find latest patchset available for same component.

1.1. SCA aks BPEL Component Current Patch set version Verification


Step1: 


Run the lsinventory command for Oracle_SOA1 home and get the installed Patch Number.

-bash-3.2$ export ORACLE_HOME=/u01/app/oracle/product/fmw/Oracle_SOA1
-bash-3.2$ cd /u01/app/oracle/product/fmw/Oracle_SOA1/OPatch
-bash-3.2$ ./opatch lsinventory -InvPtrLoc /u01/app/oracle/product/fmw/Oracle_SOA1/oraInst.loc -details



Step2: 


Now, login into support.oracle.com >> under patch & Update tab page >> provide patch number which we have got from above steps and click on search button.  


Step3: 


The below result will show us which particular patch set number this patch 19953598 is. It this case its SOA Bundle Patch 11.1.1.7.6


1.2. SOA Suite Latest Patch Set Search


In above steps we have confirmed which particular SOA Bundle PATCH set been already installed, now next steps to find out if there is any new patch set released after the one which is already installed. 

Step1: 


Login into support.oracle.com >> Patches & Update >> Product = Oracle SOA Suite >> Release = FMW 11.1.1.7.0 >> Platform = <As per your environment>
Click on search button-


Step2: 


The result which will appear will show what are all the patchset has been released. In below snap you could notice that next SOA Bundle has been release whose version is ‘SOA Bundle PATCH 11.1.1.7.8


Step3: 


Download the patch and apply in test environment, do regression test to verify all key business transactions, if it works apply into Production environment. Refer the patch readme file for detailed instructions. 

Likewise, above steps we can verify existing patch version of Oracle Service Bus (OSB) Component,  Weblogic Server component and find latest patch set released (if any) and apply into SOA environments.

1.3. Finding Weblogic Component Installed Patch Set Version 


There are two ways we can check the WLS installed product version-

Option1:


cd /u01/app/oracle/product/fmw/utils/bsu
./bsu.sh -prod_dir=/u01/app/oracle/product/fmw/wlserver_10.3 -status=applied -verbose –view



Option2: 


-bash-3.2$ export WL_HOME=/u01/app/oracle/product/fmw/wlserver_10.3
-bash-3.2$ . $WL_HOME/server/bin/setWLSEnv.sh
-bash-3.2$ java weblogic.version




Find the WLS latest Patch set in similar ways as we have found for OSB and SCA components.