Automation of Node Manager startup script !!!

Node Manager is one of key component of WLS administration. It has to be setup separately while installing OFMW binaries. This helps to centrally manage the WLS Domain servers.  Node Manager helps an administrator to centrally manage all the WLS servers in domain from WLS Admin console. 

Node Manager also help to auto restart particular WLS managed server in case WLS server become crash due to some issues. 

However, all above listed functionality can only works when Node Manager Process is running on particular vServer or physical servers.  

By default, we have to start node manager process manually using startNodeManager.shshell script. Starting process of node manager can be slightly different from implementation to implementation depending up on how it has been setup. 

Since, starting node manager is manual process, sometime we forgot this and ended-up with problem why WLS admin console is not able to communicate with all Managed servers. 

Often infrastructure guys do Database Patching and OS patching etc. wherein Physical servers or vServers need to restarted and because of this restart node manager process get dies and we need to start it manually again. 

If we are maintaining large number of vServers then startup manually node manager process takes bit longer which can be avoided completely by setting up auto startup mode.  

In below post I will try to explain how we can automate the starting process of node manager. We can write a shell script which will act as node manager service and will be having start/stop/status/restart operations. 

Once node manager service is running we can automate startup of that service using OS command. 
So, there will two steps as state below – 

1) Write a node manager service e.g. nodeMgrAutoStartSrv.sh to support start/stop/status/restart operations

2) Use the ‘chkconfig’ command to automate startup of that service. 

Node Manager nodeMgrAutoStartSrv.sh Service code

# export environment variable as per your environment e.g. 
export MW_HOME="/u01/app/oracle/product/fmw"
export JAVA_HOME="/u01/app/oracle/product/fmw/jdk"
export WL_HOME="/u01/app/oracle/product/fmw/wlserver_10.3"
export NodeManagerHome="/u01/app/oracle/product/fmw/wlserver_10.3/common/nodemanager"
export PROCESS_STRING="weblogic.NodeManager"

# setup temporary variable 
DAEMON_USER="oracle"
PROGRAM="/u01/app/oracle/admin/domain/cluster/scripts/nodeManager/startNodeManager.sh > 
$NodeManagerHome/nodemanager.out 2>&1"

# load wls environment variable by calling setWLSEnv.sh script
source $WL_HOME/server/bin/setWLSEnv.sh > /dev/null

# start function definition 
start() {
        OLDPID=`/usr/bin/pgrep -f $PROCESS_STRING`
        if [ ! -z "$OLDPID" ]; then
    echo ""
            echo "Node Manager is already running (pid $OLDPID) !"
    echo ""
            exit
        fi
echo ""
        echo "Starting Node Manager Process now :"
/bin/su $DAEMON_USER -c "$PROGRAM"
}



# stop function definition 
stop() {
        echo ""
echo "Checking Node Manager Process running Status";
echo ""
        OLDPID=`/usr/bin/pgrep -f $PROCESS_STRING`
        if [ "$OLDPID" != "" ]; then
            /bin/kill -TERM $OLDPID
    /bin/echo "Node Manager process id $OLDPID has been killed";
    echo ""
        else
            /bin/echo "Node Manager process is not running";
    echo ""
        fi
}




# restart function definition 
restart() {
        stop
        sleep 10
        start
}



#  status function definition 
status() {
OLDPID=`/usr/bin/pgrep -f $PROCESS_STRING`
        if [ "$OLDPID" != "" ]; then
        echo ""    
/bin/echo "Node Manager is running (pid: $OLDPID)"
echo ""
        else
echo ""
            /bin/echo "Node Manager is not running"
echo ""
        fi       
}

# Snap when node manager not running



# Snap when node manager is running




# switch case to decide which action has to be performed 
case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  status)
status
;;
  restart)
        restart
        ;;
 *)
        echo $"Usage: $0 {start|stop|status|restart}"
        exit 1
esac
exit 

Use the ‘chkconfig’ command to automate startup of nodeMgrAutoStartSrv.sh service 


In earlier section we have created nodeMgrAutoStartSrv.sh script. Now, next steps to configure this script for auto startup, so in the event of vServers reboot node manager process will always be up and running. 

Execute the below steps to achieve the same-

Step1: 
Copy nodeMgrAutoStartSrv.sh to this location /etc/init.d 

Note: Most Linux distributions have one thing in common. That being, all the start-up scripts are stored in the /etc/init.d/ directory.

Step2:
Change access rights and ownership

chmod 755  /etc/init.d/nodeMgrAutoStartSrv.sh
chown root:root /etc/init.d/nodeMgrAutoStartSrv.sh

Step3:
Run the below command to configure auto startup of this script

chkconfig --levels 345 nodeMgrAutoStartSrv.sh on

Note:  Above command must need to run via root user.

If interested to know more about levels 235 then please refer below link
http://serverfault.com/questions/356984/explanation-of-chkconfig-levels

Or

Step3a:

In some linux chkconfig mostly used for demean program, not for custom shell script. So, the other way to do auto startup of nodeMgrAutoStartSrv.sh script is to create manual link this script using below command.

cd /etc/rc3.d
ln -s  ../init.d/nodeMgrAutoStartSrv.sh   S99nodeMgrAutoStartSrv.sh  ;  ln -s  ../init.d/nodeMgrAutoStartSrv.sh   K99nodeMgrAutoStartSrv.sh
cd /etc/rc4.d
ln -s  ../init.d/nodeMgrAutoStartSrv.sh   S99nodeMgrAutoStartSrv.sh   ; ln -s  ../init.d/nodeMgrAutoStartSrv.sh   K99nodeMgrAutoStartSrv.sh
cd /etc/rc5.d
ln -s  ../init.d/nodeMgrAutoStartSrv.sh   S99nodeMgrAutoStartSrv.sh  ;   ln -s  ../init.d/nodeMgrAutoStartSrv.sh   K99nodeMgrAutoStartSrv.sh

Even, chkconfig command does the same thing, it create symbolic link inside rc3.d, rc4.d and rc5.d depending up run level specified in chkconfig command. While server start its looks inside these folder and start the process which resides here. There are always two process gets created for one specific process. E.g. nodeMgrAutoStartSrv.sh has created two sub process S99nodeMgrAutoStartSrv.sh and K99nodeMgrAutoStartSrv.sh which will be used start and shutting down respectively while server reboot process. 

Step4: 
Reboot the server

Step5: 
After reboot, verify whether node manager process is up and running or not.

ps -ef | /bin/grep -w weblogic.NodeManager


Or

Creating a folder path alias in Unix/Linux !!!

Creating a folder path alias in Unix/Linux !!!


In unix/linux navigating through a long folder path repetitively is bit painful task and takes a bit time when you have large number of severs to manage and perform administration task. 

In such situation creating folder path alias is the best way to create abbreviated shortcuts and change folder path quickly without navigating through whole path. 

In this very simple article instruction will be given how to create folder alias for a particular folder path and change directory easily without navigating to whole path. 

Below are the steps required to create folder alias-

1) List out the long folders path which you need to navigate frequently
2) Write down the abbreviated name for each long folder path
3) Update .bashrc and .profile file
4) Load the .profile file for particular os user.
5) Use the abbreviated value to cd to that long folder path

Step 1: 

For example purpose, I have listed out below long most used folders path for implementation – 

/u01/app/oracle/admin/domains/aserver/prd_domain
/u01/app/oracle/admin/domains/mserver/prd_domain
/u01/app/oracle/product/fmw
/u01/app/oracle/admin/domains/cluster
/u01/app/oracle/admin/domains/aserver/domain/servers/AdminServer/logs
/u01/app/oracle/admin/mserver/domains/servers
/u01/app/oracle/admin/cluster/logs

Step 2: 

Abbreviated value Long folder path

Abbreviated value
Long folder path
aserver
/u01/app/oracle/admin/domains/aserver/prd_domain
mserver
/u01/app/oracle/admin/domains/mserver/prd_domain
fmw
/u01/app/oracle/product/fmw
cluster
/u01/app/oracle/admin/domains/cluster
alog
/u01/app/oracle/admin/domains/aserver/domain/servers/AdminServer/logs
mlogs
/u01/app/oracle/admin/mserver/domains/servers
logs
/u01/app/oracle/admin/cluster/logs

Step 3:

Folder alias can be updated either directly in .profile or .bashrc file for a particular os user. The test environment where I have implemented this I have created folder alias inside .bashrc file located inside particular os user home folder. 

So, e.g. sample os user name is ‘oracle’ the .bashrc file must exist at this location /home/oracle/. If .bashrc file does not exist then admin can create this file as well using vi editor. 

Add below entries into .bashrc file

alias aserver='cd ~/../../u01/app/oracle/admin/domains/aserver/prd_domain
alias mserver='cd ~/../../u01/app/oracle/admin/domains/mserver/prd_domain
alias fmw='cd ~/../../u01/app/oracle/product/fmw
alias cluster='cd ~/../../u01/app/oracle/admin/domains/cluster
alias alog='cd ~/../../u01/app/oracle/admin/domains/aserver/domain/servers/AdminServer/logs
alias mlogs='cd ~/../../u01/app/oracle/admin/mserver/domains/servers
alias logs='cd ~/../../u01/app/oracle/admin/cluster/logs

Step 4: 

Now, update .profile file for same user to load .bashrc file
e.g.
source ~/.bashrc

Step 5: 

Now load the .profile file for ‘oracle’ user. 
Use putty to login using oracle user, after successful login, it will land ‘oracle’ user home directory, run below command 

source ~/.profile

Once you run the above command, oracle user profile will be loaded which will set all the folders path alias. 

Now, we can use the abbreviated value to access particular folder. 
e.g. to access this path ‘/u01/app/oracle/product/fmw’ just type this abbreviated value ‘fmw’ as per below snap. 

Import certificate into existing jks file for one way ssl communication

Import certificate into existing jks file for one way ssl communication

Use case described:


In this scenario, we will be adding a certificate into weblogic trust for one of the mail server from where OSB proxy service expected to polls mails.

Weblogic trust keystore is the store where weblogic keeps the certificates for those systems which are trusted systems for weblogic.

This certificate would be used while making ssl communication to those servers, its one way ssl communication.

Below are the important steps to achieve desired result -

Step1: 

Before adding any keys to existing jks file, we must need to back-up the existing jks file


Step2: 

Use the openssl utility to get the certificate from mail server.

openssl s_client -connect host:port  



Step3:

Create a new file and paste the content between BEGIN CERTIFICATE & END CERTIFICATE into that file and moved that file to your desired location at server where other jks files are stored








Step4:

Run this command to import the keys –

keytool -import -alias fujitsusmtp -keystore /u01/app/oracle/admin/aio_sit_domain/cluster/jks/mykeys.jks -file /u01/app/oracle/admin/aio_sit_domain/cluster/jks/fujitsu_smtp.txt



Step5:

Select particular WLS server from which email pooling request is supposed to get initiated e.g. WLS_OSB1, go to “keystore” tab page and provide the details of existing jks file under “turst” as below-

Keystores: changed value to “Custom Identity and Custom Trust” from default value

Custom Trust KeyStore: Specify the path of the JKS file which contains certificate Custom Trust

Keystore Type: set “JKS”

Custom Trust Keystore Passphrase: enter password which set while creating jks

Confirm Custom Trust Keystore Passphrase: repeat the password



Step6:

Add below parameter into setDomainEnv.sh file for WLS_OSB1 server to prioritize keystore location path and provide keystore credentials.

-Dssl.debug=true -Djavax.net.ssl.trustStore=CertificateLocation-Djavax.net.ssl.trustStorePassword=CertificatePassword

Step7:

Restart Admin Server and that particular WLs server for which keystore details has been added. 

Note: In osb we just need to create one proxy service, specify mail server details, create service accounts to keep credentials, this will pools message from this email server using provided credentials.





Also don’t confuse with “Service Key Provider” in OSB for one way SSL communication, its mostly been used for 2 way SSL communication where we need to generate keys in pair (public and private) and keep them inside jks file. For one way SSL communication we do need “Service Key Provider” configuration.

Reference: 

Some of the docs which help me to fix this problem -

https://docs.oracle.com/middleware/1212/wls/SECMG/identity_trust.htm#SECMG365
https://docs.oracle.com/cd/E23943_01/admin.1111/e15867/service_key_providers.htm#OSBAG983
http://theheat.dk/blog/?p=2059

Accessing Oracle Storage Cloud Service Instance


Accessing Oracle Storage Cloud Service Instance !!!


Oracle Storage cloud services is one of the mandatory service to have from Oracle Cloud Services Stack to provision any other Oracle cloud services e.g. Oracle SOA Cloud Services and Oracle Database Cloud Service. Unfortunately, Oracle does not provide any graphical tool to manage the Oracle Storage regular operation such as creating/updating/deleting containers and objects. In order to perform all these operations Oracle has recommended below three approaches -

1)      Using Rest API (cURL)
2)      Using Java Library
3)      Using Oracle Storage Cloud Software Appliance (oracle command based tool)

Fortunately, there is freeware tool “CloudBerry Explorer for OpenStack Stroage” from a company called ”CloudBerry Lab”, using which we can access Oracle Storage cloud service graphically, resulting ease of accessibility of storage and perform all sort of basic operation e.g. creating/deleting containers and objects

This post will walk through how to access of Oracle Storage Cloud Services via CloudBerry Explorer graphical tool as well as cURL command based tool.

Installing “CloudBerry Explorer for OpenStack Stroage” tool and accessing Oracle Storage cloud Service


Step1:

Download  “CloudBerry Explorer for Openstack storage” tool from below URL -



Step2:

Click on tool action from your desktop and Select “OracleCloud” storage option from new connection –





Step3:

Enter Details for your environment as describe below and click on OK button to make connection-
Connection Attributes
Example values
Display Name:
This is open string, give any name which you like for your connection e.g. “My Oracle Cloud Storage”
User Name:
Username  Syntax:- “<Oracle Storage cloud service instance id>-<IdentityDomainName>:<username>”
Example username: “Storage-jcsdemo0117:gse_cloud-admin@oracleads.com”
Password:
Your username password which we used to login into Oracle cloud service portal
Authentication Service:
Note: Check for “us6” or “us2” depending up on which data center your storage server location exists. 





Step4:

Above steps will launch tool in graphical mode wherein you can create, view and delete container and objects.




Accessing Oracle Storage Cloud Services using cURL tool


Accessing Oracle Storage Cloud service using cURL has two main steps-

Getting Authorization token

Oracle Storage Cloud Service requires authentication when executing operations against your storage service instance. Authentication is provided to the storage service instance in the form of an authentication token. You request an authentication token from the service by sending your user credentials to the service. Authentication tokens are temporary and expire in 30 minutes. This is a session time out and not an idle time out, which means that tokens expire even if they are in use. You must include your current authentication token with every operation against your service instance.

Step1:

Download curl tool from below site –






Note: Carefully select the cURL version as per your windows/ios environment. I was using win10 64 bit as result I have selected the above one.

Step2:

Change directory to the location where you have downloaded .cab file and unzipped it e.g.



Step3:

Run the below command to get the Authorization token and verify actual Storage Rest Endpoint URL –
Syntax:
curl -v -X GET \ -H "X-Storage-User: <my storage service instance name>-<my Identity  domain name>:<My Username>" \ -H "X-Storage-Pass: <my Password>" \ https://foo.storage.oraclecloud.com/auth/v1.0
Actual String:
curl -v -X GET \ -H "X-Storage-User: Storage-jcsdemo0117:gse_cloud-admin@oracleads.com" \ -H "X-Storage-Pass: XXXXXXX" \https://jcsdemo0117.storage.oraclecloud.com/auth/v1.0
In above-
my storage service instance name  is “Storage”
my Identity  domain name is “jcsdemo0117”
my Username name is “gse_cloud-admin@oracleads.com




Step4:

Once you run the above command it will show you Authorization token, make a note of this token as while creating container or object you need to pass this token in your next request.


Create Container & Objects


Step5:

Once Authorization token is obtain then we need to replace authorization token value in below string and execute put command to create the container inside storage-
Syntax:

curl -v -X PUT \
-H "X-Auth-Token: AUTH_tk209f7f2ea1265a0d3f29d28a2dc8ced6" \
-d "Hello, World!" \
https://foo.storage.oraclecloud.com/v1/myService-myIdentity3/myContainer/
myObject
Actual String-
curl -v -X PUT \ -H "X-Auth-Token: AUTH_tk05196ebb22544e4fdcffc28aa20605e0" \ -d "Hello, World!" \ https://jcsdemo0117.storage.oraclecloud.com/v1/Storage-jcsdemo0117/fpsoadbbkp/testobj



Weblogic Muxer Thread in weblogicadmintutorials.blogspot.in/


1         Weblogic Muxer Socket Thread


Weblogic has Thread group called ‘Weblogic Muxer’, once you take thread dumps you could see always 4-8 running Muxer Threads e.g. ExecuteThread: '3' for queue: 'weblogic.socket.Muxer
Muxer threads plays very important role when your Production environment struggling with performance problems.
In this article I just want to give some insight about Muxer threads which I gained from my own experience in my current project. 

1.1       What is Muxer Threads:


Muxer thread is software components which weblogic uses to read/write the data from the socket on which external system communicating, once Muxer thread reads data from socket its pass on that message to Weblogic Execute queue and from there Weblogic Self Tunning Thread pool threads pick the message and process the request.
You can unable/disable “Enable Native IO” option from Admin console for a specific servers –


1.2       Types of Muxer Threads-


There are three types of Muxer threads –

1.2.1     Native Muxers

Native muxers use platform-specific native binaries to read data from sockets. The majority of all platforms provide some mechanism to poll a socket for data. For example, Unix systems use the poll system call and the Windows architecture uses completion ports. Native muxers provide superior scalability because they implement a non-blocking thread model. When a native muxer is used, the server creates a fixed number of threads dedicated to reading incoming requests. Oracle recommends using the default setting of true for the Enable Native IO parameter which allows the server to automatically select the appropriate muxer to use.

Configuration:
Checked "Enable Native IO" only from Admin console.

Count Of Muxer Threads:
4

Stack Trace from Thread Dump:
"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon prio=3 tid=0x00000001042f4800 nid=0x1e waiting for monitor entry [0xffffffff5e6ff000]
   java.lang.Thread.State: BLOCKED (on object monitor)
                at weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:92)
                - waiting to lock <0xfffffffe20485978> (a java.lang.String)
                at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
                at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
                at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
                at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

Comment/Remark:
In this case Weblogic will use native class "DevPollSocketMuxer" for Sun hardware.

1.2.2     Java Muxer

A Java muxer has the following characteristics:
·         Uses pure Java to read data from sockets.
·         It is also the only muxer available for RMI clients.
·         Blocks on reads until there is data to be read from a socket. This behavior does not scale well when there are a large number of sockets and/or when data arrives infrequently at sockets. This is typically not an issue for clients, but it can create a huge bottleneck for a server.
If the Enable Native IO parameter is not selected, the server instance exclusively uses the Java muxer. This maybe acceptable if there are a small number of clients and the rate at which requests arrive at the server is fairly high. Under these conditions, the Java muxer performs as well as a native muxer and eliminate Java Native Interface (JNI) overhead. Unlike native muxers, the number of threads used to read requests is not fixed and is tunable for Java muxers by configuring the Percent Socket Readers parameter setting in the Administration Console. Ideally, you should configure this parameter so the number of threads roughly equals the number of remote concurrently connected clients up to 50% of the total thread pool size. Each thread waits for a fixed amount of time for data to become available at a socket. If no data arrives, the thread moves to the next socket.

Configuration:
1) UnChecked "Enable Native IO" option from Admin Console
2) Removed"-Dweblogic.MuxerClass=weblogic.socket.NIOSocketMuxer" if exist in setDomainEnv.sh or at Admin server consolehttp://docs.oracle.com/cd/E24329_01/apirefs.1211/e24401/taskhelp/tuning/EnableNIOSocketMuxer.html

Count Of Muxer Threads:
4

Stack Trace from Thread Dump:
"[ACTIVE] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x00000001035fb800 nid=0x2c runnable [0xffffffff6a5fe000]
   java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at weblogic.socket.SocketMuxer.readFromSocket(SocketMuxer.java:988)
 at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:922)
 at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:889)
 at weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:339)
 at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:252)
 at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

Comment/Remark:
Weblogic using JAVA Muxer class to read/write the data from sockets.

1.2.3     Non-Blocking IO Muxer

WebLogic Server provides a non-blocking IO implementation that may provide enhanced performance for some applications
To enable non-blocking IO:
  1. If you have not already done so, in the Change Center of the Administration Console, click Lock & Edit (see Use the Change Center).
  2. In the left pane of the console, expand Environment > Servers.
  3. On the Summary of Servers page, select the server instance for which you will enable native IO.
  4. Expand the Configuration > Tuning tab.
  5. If necessary, click Advanced to access advanced tuning parameters.
  6. Enter weblogic.socket.NIOSocketMuxer in the Muxer Class field.
  7. Click Save.
  8. To activate these changes, in the Change Center of the Administration Console, click Activate Changes.
    Not all changes take effect immediately—some require a restart
Configuration:
1)      Checked "Enable Native IO" option from Admin console.
2)       Added "-Dweblogic.MuxerClass=weblogic.socket.NIOSocketMuxer" in setDomainEnv.sh to override the Muxer class.

Count Of Muxer Threads: 4

Stack Trace from Thread Dump:
"ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" daemon prio=3 tid=0x000000010360f800 nid=0x31 runnable [0xffffffff5c2ff000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
        at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:171)
        at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0xfffffffe23f121d8> (a sun.nio.ch.Util$2)
        - locked <0xfffffffe23f121c8> (a java.util.Collections$UnmodifiableSet)
        - locked <0xfffffffe23ebc5b8> (a sun.nio.ch.DevPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
        at weblogic.socket.NIOSocketMuxer.processSockets(NIOSocketMuxer.java:320)

        at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
        at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)


Comment/Remark:
This time since we explicitly gave Muxerclass name at JVM level it’s using now "NIOSocketMuxer" class for socket reading.

Snap from Thread dump -

Refer the below URL for details understanding-

1.3       Error Related to Socket Problems –


In Weblogic .out log file you might see lots of IO Exception and Socket Exception mentioned below
<Nov 16, 2012 12:42:28 PM EST> <Error> <WliSbTransports> <BEA-381304> <Exception in HttpInboundMessageContext.close: java.io.IOException: Broken pipe
java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
        at sun.nio.ch.IOUtil.write(IOUtil.java:60)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
        Truncated. see log file for complete stacktrace

<Dec 6, 2012 1:58:11 AM EST> <Error> <WliSbTransports> <BEA-381304> <Exception in HttpOutboundMessageContext.RetrieveHttpResponseWork.run: java.io.IOException: Connection reset by peer
java.io.IOException: Connection reset by peer
                at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
                at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
                at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
                at sun.nio.ch.IOUtil.read(IOUtil.java:191)
                at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
                Truncated. see log file for complete stacktrace

1.4       General Question which rises in Mind in regard to Socket Configuration


Question1: If the configuration contains checked “Enable Native IO” and “-Dweblogic.SocketReaders=8
-Dweblogic.MuxerClass=weblogic.socket.NIOSocketMuxer”
In that case will weblogic uses native IO or non native IO.

Ans:  Above configuration will override default configuration and will use weblogic.socket.NIOSocketMuxer class which is Non blocking class been added later to provide improved performance. Refer this doc for details

http://docs.oracle.com/cd/E24329_01/web.1211/e24390/wls_tuning.htm#i1151673

Question2:  Can we use both Native IO and JAVA IO socket threads together?

Ans: no we can have only one sort of implementation, though your server configuration can have two configurations but one must be suppressed by another configuration.

Question3 : If we use Socket Reader count up to 8 using this configuration “Dweblogic.SocketReaders=8”, what will happen if we disabled the “Enable Native IO” Option, Will weblogic by default allow to use 33% of Execute Threads as Socket Reader threads?

Ans:  No, again default configuration will be override, weblogic will use 8 Muxer thread only to do socked read/write.

Question4: Which one out of Native IO and JAVA IO socket threads we should use in Production with high traffic volume?

Ans:
In our production previously we were having checked “Enable Native IO” and specified “NIOMuxerclass” and were facing many problem e.g. OSB transaction were hanging, were not getting timeout, socket timeout exception was coming in logs, later point of time as per Oracle recommendation we switch back to default Weblogic configuration which checked “Enabled Native IO”, removed NIOMuxer class and after that we witnessed lot more improvement in Production environment.

Question: What should be no of socket reader threads?

Ans:  We have tested various configuration in our Test environment and found that weblogic by default mostly use 4 Muxer threads only regardless of Native or Non Native Muxer Type. For Java Muxer document says it can use 33% threads from Execute Thread groups but we realized even with that configuration in 30 TPS load Weblogic initializing only 4 threads.

2         Summary:

As per my experience I would always suggest to go with default configuration which comes with Weblogic for Socket Reader Muxer threads, previously we have modified the configuration and introduced NIO Muxer class and also increased no of socket Reader threads but it didn’t help for any performance, even we notice lots of strange behaviour start occurring, later point of time we switch back to Default Weblogic Configuration and realized improvement. So go with default configuration. Also when you look at Thread dumps you might see Muxer Thread for Block state for longer time etc so don't be scared for that, Muxer threads are designed in such way only, its normal to see Muxer Thread Blocked state till long time.