Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J by rocker12

Performance Tuning and Troubleshooting for Oracle OC4J Version 1.0 Jurijs Velikanovs http://otn.oracle.com/ocm Introduction....................................................................................................................2 OC4J Internals ...............................................................................................................3 Physical structure .......................................................................................................3 Memory structures .....................................................................................................3 Processes ....................................................................................................................3 OC4J Troubleshooting ...................................................................................................9 Classification of Problem Areas ................................................................................9 Possible Approaches & Information Sources ..........................................................10 Java â&#x20AC;&#x153;Full thread dumpâ&#x20AC;? ..........................................................................................11 How to get Full thread dump? .............................................................................11 The interpretation of Full thread dump................................................................13 Useful Java runtime parameter ............................................................................16 Java DEADLOCK problem .................................................................................17 Log files ...................................................................................................................18 OC4J output file...................................................................................................18 OC4J default-web-access.log...............................................................................19 OS Monitoring commands.......................................................................................19 OS Tracing commands.............................................................................................20 Apache side information ..........................................................................................22 Apache Status page ..............................................................................................22 Apache log files ...................................................................................................22 Java code decompilation ..........................................................................................23 Debugging Switches ................................................................................................23 Documentation, External searches and Metalink.....................................................24 Documentation.....................................................................................................24 External searches .................................................................................................24 OC4J Performance .......................................................................................................26 Our Findings ............................................................................................................26 Java Workers........................................................................................................26 OC4J Processes....................................................................................................26 Memory................................................................................................................27 Stresstesting .........................................................................................................28 Response Time Monitoring by DMS.......................................................................29 Response Time Breakdown Example ..................................................................29 DMS Sensors Classification ................................................................................30 Collecting and Reflecting DMS Statistics ...........................................................30 Appendix A. How to diagnose opened files, sockets, pipes by OC4J process? ..........32

Introduction There is almost no information about OC4J troubleshooting and performance tuning. This paper covers available methods of OC4J performance and availability troubleshooting. The information provided is based on several years of author’s experience working with this technology. The main question paper is trying to cover is how to troubleshoot OC4J production environments, where it is almost impossible to use the traditional Java troubleshooting methods such as Java profilers because of the significant overhead. The first part of the paper describes the internals of OC4J. It is vital to understand OC4J main processes (threads) and their responsibilities to be able to troubleshoot OC4J related problems successfully. What are the main thread (process) types? What tasks are they responsible for? What are the relationships between different types of the processes? “OC4J Internals” part of this paper gives the answers to these questions. There is no any single method which would give the complete information about the problem within OC4J environment. The “OC4J Troubleshooting” section gives you the information about available methods. Use all available methods to gather the sufficient troubleshooting data for further analysis, localization of the problem area and elimination of the problem. “OC4J Performance” describes author’s main findings in the context of OC4J performance tuning and available performance monitoring capabilities. Note: As English is not the author’s native language, author apologizes for possible "syntax errors" in this document.

OC4J Internals Warning!!! Please note that all the information provided in this section is based on author’s experience working with OC4J processes. Java “Full thread dump” and Java classes decompilation were the main methods to get the information. Because of that and because the fact that Oracle Corporation doesn’t publish OC4J architecture publicly, the OC4J architecture described below possibly is not 100% accurate. Author hopes that it is correct in main terms and will be useful for reader to understand better how to troubleshoot and tune OC4J.

Physical structure OC4J physical structure (source, configuration and application files structure) is described in the Oracle documentation (See. OC4J Standalone User's Guide1). However you can find useful information about how to diagnose opened files, sockets, pipes by OC4J process in the appendix (see. Appendix A. How to diagnose opened files, sockets, pipes by OC4J process?).

Memory structures Memory structures of OC4J process are not covered in this paper in details. They are highly dependent on the application you are using. For sure there are OC4J specific memory structures for storing such information as HTTP session list, managing threads and connections pools etc. as well. But they are not covered in this paper. There will be additional paper on OC4J memory structures in the future probably.

Processes For troubleshooting and performance purposes it is extremely important to understand the architecture of OC4J instance (process). What are the main treads? What are their responsibilities? OC4J process architecture is a matter of a great interest in a context of this paper. OC4J is written entirely in Java language and it is a Java application. Not depending on the configuration you are using, JRE2 is used to run OC4J instance. The command below is used to start-up any OC4J instance: <jre location>/bin/java <1 set of parameters> -jar oc4j.jar <2 set of parameters>

Any java process is multithreaded. It is vital to know multithreading concepts for understanding OC4J internals. Multithreading is the way how many tasks (threads) can be executed within a single java process. Threads are executed in the same context and have access to all process memory structures. Different operational systems have different implementations of multithreading within the single process. In most of the operational systems you will see OC4J as single process. There are different possibilities on different OS to monitor threads within the single process. Sun Solaris for example realizes multithreading through light-weight processes. Light-weight processes can be monitored easily. You can use prstat -Lp <pid> to see CPU utilization by different threads within the single process. Unlike most of the OS 1

The physical structures of OC4J is the mostly the same despite OC4J version. Author recommends to read “Oracle® Application Server Containers for J2EE Standalone User's Guide, 10g Release 2 (10.1.2), Part No. B14361-01” chapter “Advanced Configuration and Development” 2 There is JDK used actually, OC4J instance for JSP executing compile any JSP to an *.java file and *.calss file afterwards.

Linux realizes multithreading by running each process thread as dedicated process. You can monitor each thread using regular OS utilities such as ps, strace, pstree etc. There is an OC4J single process on the Figure 1 “OC4J instance” below. The OC4J instance consists of the processes or threads (quadrangles) and memory structures (ovals).

ONS

PORT: 3001

Thread OC4JMonitor Thread

new req

AJP Connection Listener

ApplicationServerThread

Thread

ApplicationServerThread

ApplicationServerThread ApplicationServerThread

Java native

ApplicationServerThread ...

AJP 1.3

ONet

ApplicationServerThread ApplicationServerThread

Memory Structures Application objects

3101

RMI RMIServer Server

3201

Http Session List

JMSServer

PORT: 3101

PORT: 3201

...

TaskManager

TaskManager JMSServer JMSServer JMSServer JMSServer Thread Thread

JDBC Connection pool

Java native VM Thread VM Thread Signal Dispatcher Signal Dispatcher Suspend Checker Suspend ThreadChecker Thread

VM Periodic Task VMThread Periodic Task Thread Finalizer Finalizer Reference Reference Handler Handler

Figure 1 “OC4J instance”

Each group of processes is responsible for different tasks executing within OC4J instance. Let’s go through OC4J process groups and describe their responsibilities: Java native threads Group processes (Java HotSpot 1.3.1): VM Thread, VM Periodic Task Thread, Reference Handler, Signal Dispatcher, Suspend Checker Thread, Finalizer This group of the threads exists in any java application process. They are initialized during a start-up of java application and are responsible for generic java routines, such as: initializing and terminating other threads, freeing unused memory etc. OC4J Listener – threads Group processes: HTTP/AJPConnectionListener, RMIServer (2), JMSServer

These threads handle incoming client requests coming through different protocols. In case of Oracle AS configuration, you can use command listed below to detect ports used by listener processes for each started OC4J process: $ opmnadmin debug comp=pm ... ======== PM ======== PM requests processed: 7 PROCESS TABLE UID ---------750407 619335 488263 226119 160583 95047 $

PID ----20502 4535 4533 15875 15513 15456

FLAGS TYPE STATUS REF HTTP AJP RMI JMS -------- ------- ------- ------ ----- ----- ----- ----00000040 OC4J Alive 1 0 3002 3103 3203 00000040 OC4J Alive 1 0 3003 3104 3204 00000040 OC4J Alive 1 0 3001 3102 3202 00000040 OC4J Alive 1 0 3000 3101 3201 00000040 Apache Alive 1 7780 0 0 0 00000040 Generic Alive 1 0 0 0 0

HTTP/AJPConnectionListener – depending on configuration you will have one of the listeners (HTTP or AJP). HTTPConnectionListener appears if you run standalone OC4J process. You can set port on which this process accepting requests in http-web-site.xml file: OH/j2ee/<comp_name>/config/http-web-site.xml <web-site port="8888" ...> ... </web-site>

AJPConnectionListener is the main listener process in the typical Oracle AS configuration. Normally3 the most requests come in through this process. It accepts clients’ requests forwarded by Apache child processes through AJP protocol. AJP – Apache JServ Protocol4. A AJP port negotiated during OC4J process start up. The OPMN process informs Apache mod_oc4j component on AJP chosen port. You can set the range of the AJP ports in opnm.xml file5 for each OC4J component: OH/opmn/conf/opmn.xml <oc4j ... > ... <port ... ajp="3001-3101" .../> ... </oc4j>

HTTP/AJPConnectionListener process fulfils the default-web-access.log file. OH/j2ee/<comp_name>/log/<comp_name>_default_island_<proc_num>/default -web-access.log

This file is the analogue of the Apache’s accecc_log log file where OC4J process writes information about each clients’ request. HTTP/AJPConnectionListener process doesn’t process clients’ requests. It accepts and assigns each request to available ApplicationServerThread process for further processing. After requests had been assigned ApplicationServerThread communicates with Apache child process directly.

Unless you run OC4J for EJB calls processing See http://jakarta.apache.org/tomcat/connectors-doc-archive/jk2/common/AJPv13.html#Intro for more information about AJP. 4

RMIServer (2) – There are two processes. First one accepts clients’ RMI requests through public network interfaces (external IP addresses). Second listener process is mainly used for administrating requests processing. Independently of configuration you are using (standalone or Oracle AS) the OC4J process is listening on 127.0.0.1:RMIport for shutdown/reload and other administration commands. Even if you are using Oracle AS configuration, a OMPN calls admin.jar to contact OC4J process through RMI port. For example the command issued by the OPMN to shutdown OC4J process listed below6: /u01/app/oracle/product/J2EE01/jdk/bin/java -DOPMN=true -Doracle.home=/u01/app/oracle/product/J2EE01 -jar admin.jar ormi://localhost:3104 admin null -shutdown

Public RMI listener accepts RMI requests to call methods of the objects located inside OC4J. In general terms this protocol enables to make distributed java calls to objects located inside OC4J. Description of RMI protocols is out of the scope of this paper. As AJP listener thread RMIServer thread doesn’t process calls. It just assigns request to available ApplicationServerThread threads. Used RMI ports are negotiated7 during the start-up of the OC4J process. You can set a range of ports in the opmn.xml file: OH/opmn/conf/opmn.xml <oc4j ... > ... <port ... rmi="3101-3200".../> ... </oc4j>

JMSServer – It is like RMI listeners from technical point of view. Java Message Service (JMS) is one of the required protocols by J2EE specification. JMS enables to send and receive data and events between containers asynchronously. ApplicationServerThread ApplicationServerThread – is a group of threads which processes users’ requests. Basically these treads execute application code, compile and execute JSP, communicate with database and send a response to the Apache child processes. They are workers who are doing most the job within OC4J container. The worker threads are linked up to the thread pool managed by TaskManager thread. Listeners assign users’ requests to the available ApplicationServerThread threads. You can define minimal and maximal number of the worker threads in server.xml file. OH/j2ee/<comp_name>/config/server.xml <application-server ...> ...

You can see it in the Process Manager log file OH/opmn/logs/ipm.log with 6 logging level switched on. You can change the PM logging level in the OH/opmn/conf/opmn.xml file: <log-file path="OH/opmn/logs/ipm.log" level="6"/> PS Logging overhead is minimal. 7 Unless you use stand alone configuration then port is configured in rmi.xml

<global-thread-pool min="100" max="300" queue="30" keepAlive="-1" cxmin="100" cx-max="300" cx-queue="30" cx-keepAlive="-1" debug="true"/> ... </application-server>

min, max, queue, keepAlive parameters are not mandatory. If you do not set them OC4J creates a single ApplicationServerThread in the thread pool during the start-up and adds additional threads on an as-needed basis. Threads allocation is an expensive task. OC4J spends CPU resource to allocate/deallocate worker threads. You can preallocate threads in the thread pool by setting min parameter for better performance. OC4J allocates min threads during the start-up. TaskManager never reduces the number of threads in the pool less then min value during OC4J life time. If there is no available thread in the pool OC4J puts new request in the queue. In case the queue length reaches a limit set by queue parameter OC4J spawns a new ApplicationServerThread thread. OC4J newer creates more threads then set by max parameter. If thread is in an idle state longer then keepAlive parameter value (in milliseconds) TaskManager removes the thread from the pool. If you set debug parameter to true TaskManager process prints the thread pools information every time of the execution. 1108429632732:ThrP[100] ThrQ[0] 1108429632732:CxP[100] CxQ[0] - WebC[0] RmiC[0] 1108429633742:ThrP[100] ThrQ[0] 1108429633742:CxP[100] CxQ[0] - WebC[0] RmiC[0] 1108429634752:ThrP[100] ThrQ[0] 1108429634752:CxP[100] CxQ[0] - WebC[0] RmiC[0]

TaskManager TaskManager process is responsible for regular tasks execution. You can change the default execution frequency (1000 = 1 sec) by setting parameter in the server.xml file: <application-server ...

taskmanager-granularity="10000" ...>

In addition to thread pool housekeeping task TaskManager has other tasks. TaskManager for example is responsible for removing expired http and JDBC sessions. You can monitor different tasks statistics using dmstool utility: $ dmstool -table oc4j_task | grep 3001 | grep completed .../TASK[1]_com.evermind.server.ApplicationServerTask/run().completed 3811 .../TASK[2]_com.evermind.server.jms.JMSTask/run().completed 3811 ops .../TASK[3]_com.evermind.server.rmi.RMITask/run().completed 3811 ops .../TASK[4]_com.evermind.server.rmi.RMITask/run().completed 3811 ops .../TASK[5]_default/run().completed 3811 ops .../TASK[6]_com.evermind.sql.DriverManagerXADataSource/run().completed 3811 .../TASK[7]_com.evermind.sql.OrionPooledDataSource/run().completed 3811 .../TASK[8]_com.evermind.sql.OrionCMTDataSource/run().completed 3811 ops .../TASK[9]_BC4J/run().completed 3811 ops .../TASK[10]_PANDA/run().completed 3811 ops .../TASK[11]_com.evermind.server.http.SessionTimeoutTask/run().completed 3811 $

ops

ops ops

ops

Note: There is another set of parameters with prefix cx-. Documentation refers to this set of parameters as parameters for second type of pool â&#x20AC;&#x201C; conection pool. Author was unable to find practical differences between these two types of pools. OC4JMonitorThread+ OC4JMonitorThread and two Thread type treads are responsible for a communication with ONS (Oracle Notification Service) process. One of Thread type treads makes

connection to the ONS process immediately after container start-up. Those threads are responsible for sending and receiving notifications from other Oracle AS processes.

OC4J Troubleshooting The troubleshooting in OC4J environment is connected strongly to a performance tuning: often the root cause of the performance problems is serialization (bugs) introduced in the container, framework or application code, often users claim that OC4J system hangs when there are the performance problems within a database tier. It is vital to have and to understand the available troubleshooting methods for implementing and maintaining OC4J based systems successfully. Using information and troubleshooting methods described in this paper you will be able to identify the areas of the problems, to gather all the necessary information and to provide it to the developers of the applications or to Oracle Support. In most cases there is necessity to correct the code of the application, used framework or even OC4J. So, it is extremely important to localize the problem area, gather all necessary information and provide it to the appropriate organization.

Classification of Problem Areas At the beginning of troubleshooting process it is important to be aware of possible types of problems. Figure 2 highlights possible problem areas within OC4J environments.

OC4J JVM(java) PORT: 3001

AJP JMSServer JMSServer Connection Listener

Apache

New Request as sig n

ApplicationServerThread ApplicationServerThread ApplicationServerThread ApplicationServerThread ApplicationServerThread TaskManager

ApplicationServerThread

ApplicationServerThread ApplicationServerThread ApplicationServerThread

JDK OS

Figure 2

1. OC4J container inside. This type of problems can be introduced by infinitive loops, suboptimal application code, bugs of underlying software, serialization introduced within used algorithms etc. The threads8 can wait on the objects locked by other system of application threads. The Java deadlocks situation is possible within a single OC4J process. The problems of this type can have the spinning characteristics, when the thread consumes a lot of CPU resources, as hanging ones (when the thread is waiting on some locked resources). 2. Interface to Apache processes. Sometimes threads can hang due to problems with interface to Apache processes. The situation when OC4J thread and 8

In a context of this section author uses terms â&#x20AC;&#x153;application threadâ&#x20AC;? or thread to reference ApplicationServerThreads (see. OC4J internals) which are worker threads within OC4J environment.

Apache child process send the AJP messages to each other in the infinitive loop can happen. The threads can have problems while receiving or sending the information to Apache processes. 3. Interface to the database. Often the database problems (such as enqueues, long running queries or DML operations, parsing or hanging cases) are interpreted as OC4J problems. It is important to have the possibilities to troubleshoot and to identify problem areas. 4. 500 Internal Errors or OC4J unavailability. Unlike previous types of problems this one can be identified easily. If there are problems with AJP (HTTP, RMI, JMS) Listener thread the whole OC4J process is not available for application users’ requests. Users will receive “500 Internal Error” message on the screens of the browsers. The problems of this type as well as described above will be useful in further investigation and troubleshooting.

Possible Approaches & Information Sources There are no many methods of OC4J process troubleshooting. No one of them will give full information about your problem. You have to use a combination of methods to diagnose OC4J related problems successfully. You can identify the most of CPU consuming threads using OS Monitoring commands for example when use Java “Full thread dump” to get an information of code to be executed and navigated to the source of problem by using Java code decompilation. Author identified and successfully used the methods and the additional information sources listed below: •

Java “Full thread dump” – often underestimated method for Java processes troubleshooting. This is the best source of troubleshooting information for Java processes. For example, Java “Full thread dump” can give the information of why the OC4J process is no longer responding. This method is extremely effective in combination with OS monitoring commands. It is vital to understand Java “Full thread dump” for OC4J troubleshooting. You do not have to be a Java developer to make a preliminary analysis of dumps and to make the conclusions. This method is described in this paper in more details as well as other methods listed.

•

Log files – there is a couple of useful OC4J log files you can use for troubleshooting purposes.

•

OS Monitoring commands – the OS commands are effective in the combination with Java “Full thread dump”. You identify top resources consuming Java threads by OS commands, then investigate OC4J process Java dump to determine the state of the tread and executed code. You can use also a reverse process to highlight the suspicious threads by analyzing Java “Full thread dump” to monitor such threads by OS commands in order to get knowledge of the resource consumption.

•

OS Tracing commands – in the same time you can diagnose problems by using this type of troubleshooting. You can use OS Tracing to attach to the suspicious thread and to see thread’s outgoing calls to the OS. This method is also useful in case you are unable to get Java “Full thread dump”.

•

Apache side information – Apache monitoring tools and log files can give you the indirect information about the state of OC4J processes.

•

Java code decompilation – you can decompile Java code for troubleshooting purposes

•

Documentation, External searches and Metalink – other useful OC4J troubleshooting information sources are covered also in this paper.

As there are no many OC4J troubleshooting methods available, you have to use all of them in combination to be successful in your troubleshooting efforts. Ensure automatic information gathering and publishing in case of problems. Make this information available to the developers and to the support organization representatives for the effective problem solving process.

Java “Full thread dump” Most of JRE/JDK environments9 have built-in mechanism to dump a snapshot of all Java threads’ states. This dump is called full thread dump. For each Java thread, you get a Java thread’s name, a thread’s ID, and a Java call stack of the thread. Java-level deadlocks information is also displayed in the dump. This section explains how to get and to interpret full thread dumps of the OC4J process.

How to get Full thread dump? There are four steps you have to execute to get OC4J Java “Full thread dump”: 1. Get OC4J process PID (OS process identification) 2. Locate OC4J output file 3. Send SIGQUIT (Quit from keyboard) signal to the process 4. Check the output file Getting OS PID of OC4J You can use ps, pstree or opmnadmin 10utilities to get OC4J process OS PID. $ opmnadmin debug comp=pm HTTP/1.1 200 OK Content-Length: 677 Content-Type: text/html Response:

======== PM ======== PM requests processed: 5 PROCESS TABLE UID ---------456236 390700 521772 259628 194092 128556 9

PID ----13512 13511 15874 13415 13338 13281

FLAGS TYPE STATUS REF HTTP AJP RMI JMS -------- ------- ------- ------ ----- ----- ----- ----00000040 OC4J Alive 1 0 3003 3104 3204 00000040 OC4J Alive 1 0 3002 3103 3203 00000040 OC4J Alive 1 0 3001 3102 3202 00000040 OC4J Alive 1 0 3000 3101 3201 00000040 Apache Alive 1 7780 0 0 0 00000040 Generic Alive 1 0 0 0 0

Java HotSpot version from Sun Microsystems used by all versions of OC4J and hashave Java “Full thread dump” build-in functionality. 10 opmnadmin is Oracle AS utility. Starting from 9.0.4 version of Oracle AS it is renamed to opmnctl. Output format differs in deferent Oracle AS versions.

opmnadmin command displays PID-s as well as other useful information about each component managing bu OPMN process. Locating OC4J output file Use ps or command like that for your OS to find out the output file name: $ ps -fw --width 4000 -p 16114 UID PID PPID C STIME TTY TIME CMD oracle 15874 15902 0 23:40 ? 00:00:00 /u01/app/oracle/product/J2EE01/jdk/bin/i386/native_threads/Java -Xms512M -Xmx512M Doracle.ons.oraclehome=/u01/app/oracle/product/J2EE01 Doracle.ons.oracleconfighome=/u01/app/oracle/product/J2EE01 -Doracle.ons.clusterid=1 Doracle.ons.instanceid=10.130.2.9.20dcd9.fdc4ac6949.-8000 Doracle.ons.indexid=PANDY.default_island.1 -DOPMN=true -jar oc4j.jar -config /u01/app/oracle/product/J2EE01/j2ee/PANDY/config/server.xml -properties -instance vmId C_10.130.2.9.20dcd9.fdc4ac6949.8000#J2EE01.ap1.voava.lv#521772#PANDY#PANDY#default_island -ajp 3001 -rmi 3102 -jms 3202 $ ps -wo args --width 4000 -p 16114 | grep -v COMMAND | awk -F"oracle.ons.indexid=" '{print $2}' | awk '{print $1}' PANDY.default_island.1

In the example above the output file name is PANDY.default_island.1, where PANDY is an OC4J instance name, 1 is an index of the OC4J process and island is a name of an island. Typically the output file is located in OH/opmn/logs directory. Sending SIGQUIT signal Execute kill -3 <pid> command to send SIGQUIT (Quit from keyboard) signal to the JVM. As Java “Full thread dump” is a generic JRE functionality, the JVM which runs the OC4J process spools dump to the standard output file you located in previous step. Check the output file Check the log file. You will see output like provided below. Full thread dump: "ApplicationServerThread-5" prio=1 tid=0x8450488 nid=0x576d waiting on monitor [0x6877f000..0x6877f870] at Java.lang.Object.wait(Native Method) - waiting on <437be538> (a EDU.oswego.cs.dl.util.concurrent.LinkedNode) at EDU.oswego.cs.dl.util.concurrent.SynchronousChannel.poll(SynchronousChannel.Java:353) - locked <437be538> (a EDU.oswego.cs.dl.util.concurrent.LinkedNode) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.Java:767) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.Java:801) at Java.lang.Thread.run(Thread.Java:479) "Thread-10" prio=1 tid=0x8162568 nid=0x5738 waiting on monitor [0..0xbffe9b08] "OC4JMonitorThread" daemon prio=1 tid=0x83c00b0 nid=0x576a waiting on monitor [0x698ff000..0x698ff870] at Java.lang.Object.wait(Native Method) - waiting on <436e2b28> (a Java.lang.Object) at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.Java:210) - locked <436e2b28> (a Java.lang.Object) at oracle.ons.NotificationQueue.dequeue(NotificationQueue.Java:183) at oracle.ons.Subscriber.receive(Subscriber.Java:136) at com.evermind.server.OC4JMonitorThread.run(OC4JMonitorThread.Java:419) at Java.lang.Thread.run(Thread.Java:479) ... "VM Thread" prio=1 tid=0x80821e8 nid=0x4dd runnable "VM Periodic Task Thread" prio=1 tid=0x808cb98 nid=0x4e0 waiting on monitor "Suspend Checker Thread" prio=1 tid=0x808d518 nid=0x4e1 runnable

Java “Full thread dump” begins with a header Full thread dump. There is a list of all the threads after the header.

The interpretation of Full thread dump As mentioned above Java “Full thread dump” begins with the header, the list of all the threads follows. Optionally there is Java-level deadlocks information if detected by the JVM at the end of the threads dump. This is an example of the available information about the single thread in the dump: "OC4JMonitorThread" daemon prio=1 tid=0x67691ec8 nid=0x3ef2 waiting on monitor [0xa53ff000..0xa53ff89c] at Java.lang.Object.wait(Native Method) at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.Java:251) at oracle.ons.NotificationQueue.dequeue(NotificationQueue.Java:224) at oracle.ons.Subscriber.receive(Subscriber.Java:136) at com.evermind.server.OC4JMonitorThread.run(OC4JMonitorThread.Java:419) at Java.lang.Thread.run(Thread.Java:479)

The most interesting parts are the following: • OC4JMonitorThread – thread’s name assigned by Java application (by OC4J in our case). OC4J assigns the unique meaningful names to the threads at the moment when they are created. Thread.currentThread().setName During the creation of the thread Java procedure is called for this purpose. If application doesn’t call mentioned function JVM assigns Thread-XX name to the thread. • nid=0x3ef2 – thread’s ID in a hexadecimal notation. This ID corresponds to OS thread’s ID. This way you can map the thread dump information to the threads you can monitor using OS utilities. It is very important to have such a possibility. You can get the complete information about particular Java thread in case of problems. • waiting on monitor - execution state of the thread. The typical threads’ states are the following: runnable, waiting on monitor, waiting for monitor entry. In this document in the context of OC4J there is more detailed description of each state. • oracle.ons.NotificationQueue.internalDequeue – the last part is the current stack trace snapshot of the thread. You can see all functions called by the thread. The most upper part of the stacks corresponds to the current Java function executed by the thread. In most cases you can understand what is the thread doing using the stack information and can find the way for further investigation of the problem.

Different types of the threads within OC4J container In most cases you will see the following threads’ types executing OC4J applications: OC4J Listeners - AJPConnectionListener, HTTPConnectionListener, RMIServer(2), JMSServer OC4J Workers - ApplicationServerThread Other OC4J - OC4JMonitorThread, TaskManager, Thread JAVA native - VM Thread, VM Periodic Task Thread, Reference Handler, Signal, Dispatcher, Suspend Checker Thread, Finalizer Each thread’s type is described in the “OC4J Internals” part of this document.

Different states of the threads In Java “Full thread dump” there is the information about the state of each thread. Author has seen three states of the threads so far: • runnable – this state indicates that the thread executes an application code or is waiting to receive a data from the socket. This state is typical for all OC4J

Listeners threads. If some of Listeners threads are in other state for a longer time you probably have a problem and have to investigate it. Workers threads in this state either execute Java application code or waiting on the socket for the data. For example worker thread can wait on database query to be executed. You can identify easily what is doing the thread looking on current thread’s stack. •

waiting on monitor - waiting for some event or notification to occur. Often in OC4J this state indicates that the thread is idle – thread is waiting for a task to be executed. Currant call (shown in java stack) for thread in this state usually is Java.lang.Object.wait(Native Method) that correspond to wait() or sleep() functions in used in the Java code. This state is typical for all non Listener threads in idle or sleeping state. Worker threads in this state are idle and available to assign a task. TaskManager thread is waiting in this state for the next execution time (calls sleep() function).

•

waiting for monitor entry – thread in this state is waiting for a object locked by the another thread. This is not normal state for the thread. If there are many threads waiting on the same object this indicates clearly a bottle neck in the system. You have to investigate this situation. Note the threads’ stacks, this will show you the place in the code where the problem occurs; note all the information available for the thread holding the lock.

Some threads examples. Example 1 "OC4JMonitorThread" daemon prio=1 tid=0x679a3518 nid=0x573 waiting on monitor [0xa53ff000..0xa53ff89c] at Java.lang.Object.wait(Native Method) at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.Java:251) at oracle.ons.NotificationQueue.dequeue(NotificationQueue.Java:224) at oracle.ons.Subscriber.receive(Subscriber.Java:136) at com.evermind.server.OC4JMonitorThread.run(OC4JMonitorThread.Java:419)

This is the typical state for OC4JMonitorThread OC4J thread. This thread is waiting for a work to be processed. Java stack indicates that this thread executes ONS (Oracle Notification Service) connected routines. Example 2 "Thread-9" at at at at at

daemon prio=1 tid=0x67a02920 nid=0x571 runnable [0xa57ff000..0xa57ff89c] Java.net.SocketInputStream.socketRead(Native Method) Java.net.SocketInputStream.read(SocketInputStream.Java:85) oracle.ons.InputBuffer.readMoreData(InputBuffer.Java:267) oracle.ons.InputBuffer.getNextString(InputBuffer.Java:222) oracle.ons.ReceiverThread.run(ReceiverThread.Java:225)

The thread is waiting to receive the data from the ONS process. This thread connects to the ONS process immediately after OC4J starts-up and is waiting for the messages. As soon as message arrives the thread assigns OC4JMonitorThread thread to process this message. Example 3 "ApplicationServerThread-5" prio=1 tid=0x6794da88 nid=0x43f waiting on monitor [0xb15ff000..0xb15ff89c] at Java.lang.Object.wait(Native Method) at EDU.oswego.cs.dl.ut...concurrent.BoundedBuffer.poll(BoundedBuffer.Java:170) at EDU.oswego.cs....concurrent.PooledExecutor.getTask(PooledExecutor.Java:768) at EDU.oswego....concurrent.PooledExecutor$Worker.run(PooledExecutor.Java:802) at Java.lang.Thread.run(Thread.Java:479)

This is the typical state for ApplicationServerThread OC4J thread. This thread is waiting for the work to be processed. OC4J Listener threads assign work to idle worker threads. Example 4 "AJPConnectionListener [0.0.0.0/0.0.0.0:3001]" prio=1 tid=0x67911828 nid=0x439 runnable [0xb19ff000..0xb19ff89c] at Java.net.PlainSocketImpl.socketAccept(Native Method) at Java.net.PlainSocketImpl.accept(PlainSocketImpl.Java:463) at Java.net.ServerSocket.implAccept(ServerSocket.Java:238) at Java.net.ServerSocket.accept(ServerSocket.Java:217) at com.evermind...http.AJPConnectionListener.run(AJPConnectionListener.Java:58) at EDU.oswego.c...concurrent.PooledExecutor$Worker.run(PooledExecutor.Java:804) at Java.lang.Thread.run(Thread.Java:479)

AJPConnectionListener is one of the OC4J listener threads. The state shown above is typical for any OC4J Listener thread. Call to the AJP function shown in the thread Java stack indicates that this thread is listening on the APJ port to accept new connections. It assigns any received message addressed to available ApplicationServerThread thread to put it to the queue. If this thread is in another state for a longer (more then for a couple of seconds) time this is a clear indication of the problem, you have to investigate it. Example 5 "ApplicationServerThread-77" prio=1 tid=0x8391048 nid=0x4a75 runnable [0xb4dff000..0xb4dff89c] at Java.net.SocketInputStream.socketRead(Native Method) at Java.net.SocketInputStream.read(SocketInputStream.Java:85) at com.evermind.io...nputStream.readChunk(SingleReadBufferInputStream.Java:116) at com.evermind.io...ufferInputStream.read(SingleReadBufferInputStream.Java:35) at com.evermind.server.http.AJPRe...ler.initRequest(AJPRequestHandler.Java:345) at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.Java:131) at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.Java:72) at EDU.oswego.c...concurrent.PooledExecutor$Worker.run(PooledExecutor.Java:804) at Java.lang.Thread.run(Thread.Java:479)

It is a problem possible to occur on Apache interface. Worker thread is reading or waiting for the information to be received from the socket. The stack trace indicates clearly that the ApplicationServerThread thread is reading AJP data from the socket connected to Apache child process. It is not always indicating the problem. However if the thread utilizes the CPU resource for a longer time this obviously is the problem. Convert nid=0x4a75 to the decimal value (19061) and monitor the thread from the OS. If you are on Linux you can use ps command to monitor whether the thread is utilizing CPU constantly: $ while [ 1 ] ; do ps -p 13285 -o pcpu | grep -v CPU ; sleep 1 ; done 25.3 25.3 25.3

See “Error! Reference source not found.” and “Error! Reference source not found.” for more details. Example 6 "ApplicationServerThread-71" prio=1 tid=0x8379130 nid=0x4a6f runnable [0xb59fe000..0xb59ff89c] at Java.net.SocketInputStream.socketRead(Native Method) at Java.net.SocketInputStream.read(SocketInputStream.Java:85) at oracle.net.ns.Packet.receive(Unknown Source) at oracle.net.ns.DataPacket.receive(Unknown Source) at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.Java:931)

at at at at at at at at at at at at at at at at at at at at at at at at at at at at at at at at at at at at at at at at

oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.Java:893) oracle.jdbc.ttc7.Oall7.receive(Oall7.Java:375) oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.Java:1986) oracle.jdbc.ttc7.TTC7Protocol.parseExecuteFetch(TTC7Protocol.Java:1144) oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.Java:2484) oracle.jdbc.driver...atement.doExecuteWithTimeout(OracleStatement.Java:2850) oracle.jdbc.driver...atement.executeUpdate(OraclePreparedStatement.Java:609) oracle.jdbc.driver...tatement.executeQuery(OraclePreparedStatement.Java:537) oracle.jbo.server.ViewObjectImpl.getQueryHitCount(ViewObjectImpl.Java:1522) oracle.jbo.server...llection.getEstimatedRowCount(QueryCollection.Java:1530) oracle.jbo.server...owSetImpl.getEstimatedRowCount(ViewRowSetImpl.Java:1482) oracle.jbo.server...bjectImpl.getEstimatedRowCount(ViewObjectImpl.Java:4596) _common._comp._CMN__...ent._jspService(_CMN__DataScrollerComponent.Java:115) com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.Java:56) oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.Java:317) oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.Java:476) oracle.jsp.runtimev2.JspServlet.service(JspServlet.Java:390) Javax.servlet.http.HttpServlet.service(HttpServlet.Java:853) com.evermind.server.http...FilterChain.doFilter(ResourceFilterChain.Java:65) oracle.security.jazn.oc4j.JAZNFilter.doFilter(JAZNFilter.Java:283) com.evermind.server.http...patcher.invoke(ServletRequestDispatcher.Java:566) com.evermind.server.http...atcher.include(ServletRequestDispatcher.Java:119) com.evermind.server.http...r.include(GetParametersRequestDispatcher.Java:95) com.evermind.server.http...PageContext.include(EvermindPageContext.Java:280) oracle.jbo.html.jsp.datatags.ComponentTag.doStartTag(ComponentTag.Java:70) _skd._d001._D2260F__Browse._jspService(_D2260F__Browse.Java:249) com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.Java:56) oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.Java:317) oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.Java:476) oracle.jsp.runtimev2.JspServlet.service(JspServlet.Java:390) Javax.servlet.http.HttpServlet.service(HttpServlet.Java:853) com.evermind.server.http...FilterChain.doFilter(ResourceFilterChain.Java:65) oracle.security.jazn.oc4j.JAZNFilter.doFilter(JAZNFilter.Java:283) com.evermind.server.http...patcher.invoke(ServletRequestDispatcher.Java:566) com.evermind.server.http...orwardInternal(ServletRequestDispatcher.Java:306) com.evermind.server.http...ndler.processRequest(HttpRequestHandler.Java:767) com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.Java:166) com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.Java:72) EDU.oswego.c...concurrent.PooledExecutor$Worker.run(PooledExecutor.Java:804) Java.lang.Thread.run(Thread.Java:479)

The problem on the DB interface is possible. Worker thread is reading or waiting for the information from the database. It is clear from the stack that this thread is waiting for a response from the database. You can see the name of Java server page which made the call to the database _skd._d001._D2260F__Browse. If the thread is in this state for a longer time you have to look on the database tier to investigate the reason of a slow response time. If there are many threads waiting on the database it is the bottleneck in your environment.

Useful Java runtime parameter It is difficult to investigate the threads in waiting for monitor entry state. JVM doesnâ&#x20AC;&#x2122;t reflect information about locked objects by default. Take a look on the example below. "ApplicationServerThread-18" prio=1 tid=0x692c40e0 nid=0x2f92 waiting for monitor entry [0xbc1ff000..0xbc1ff89c] at oracle.jbo.server.ViewRowSe...setRangeSize(ViewRowSetIteratorImpl.Java:376) at oracle.jbo.server.ViewRowSetImpl.setRangeSize(ViewRowSetImpl.Java:1659) at oracle.jbo.server.ViewObjectImpl.setRangeSize(ViewObjectImpl.Java:4626) at lv.vovaa.panda.common.da...DataSourceCmn.doStartTag(DataSourceCmn.Java:189) at _nsk._nsk001._M2801F__Edit._jspService(_M2801F__Edit.Java:150) at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.Java:56) at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.Java:317) at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.Java:476) at oracle.jsp.runtimev2.JspServlet.service(JspServlet.Java:390) at Javax.servlet.http.HttpServlet.service(HttpServlet.Java:853)

There is a problem inside the OC4J container. Worker thread is waiting on an object blocked by the other thread. If the thread is in this state for a while or there are many

threads in this state this indicates clearly the problem in the application code. You are unable to find the blocking thread and locked object using the information provided. There is a very helpful and often underestimated Java generic parameter “-XX:+JavaMonitorsInStackTrace”. After you set this parameter in Java-option section of the opmn.xml file the JVM will reflect information about locked objects by the threads the dump: "ApplicationServerThread-96" prio=1 tid=0x69d6c8c0 nid=0xdce waiting for monitor entry [0xb25ff000..0xb25ff89c] at oracle.jbo.server.ViewRowSetIteratorImpl.setRangeSize(ViewRowSetIteratorImpl.Java:376) - waiting to lock <460e0338> (a oracle.jbo.JboSyncLock) at oracle.jbo.server.ViewRowSetImpl.setRangeSize(ViewRowSetImpl.Java:1659) at oracle.jbo.server.ViewObjectImpl.setRangeSize(ViewObjectImpl.Java:4626) at lv.vovaa.PANDY.common.da...DataSourceCmn.doStartTag(DataSourceCmn.Java:189) - locked <45f722d0> (a lv.vovaa.PANDY.common.datatags.DataSourceCmn) at _nsk._nsk...Tickets__browse._jspService(_AP3220F__Tickets__browse.Java:105) at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.Java:56) at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.Java:317) at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.Java:476) at oracle.jsp.runtimev2.JspServlet.service(JspServlet.Java:390) at Javax.servlet.http.HttpServlet.service(HttpServlet.Java:853)

As you can see the JVM reflects the information about the objects locked by the thread (object_id=45f722d0) as well as about the object which the thread is waiting on (object_id=460e0338). It is possible to locate the thread which locked the object using the information provided: "ApplicationServerThread-55" prio=1 tid=0x69d512b8 nid=0xda4 runnable [0xb77fe000..0xb77ff89c] at Java.net.SocketInputStream.socketRead(Native Method) at Java.net.SocketInputStream.read(SocketInputStream.Java:85) at oracle.net.ns.Packet.receive(Unknown Source) ... at oracle.jbo.server.QueryCollection.executeQuery(QueryCollection.Java:549) at oracle.jbo.server.ViewObjectImpl...yForCollection(ViewObjectImpl.Java:2600) - locked <460e0338> (a oracle.jbo.JboSyncLock) at lv...PboViewObjectImpl.pboExecuteQueryForCollection(PboViewObject.Java:324) at lv...PboViewObjectImpl.executeQueryForCollection(PboViewObjectImp.Java:309) at oracle.jbo.server.ViewRowSetImpl.execute(ViewRowSetImpl.Java:521) ...

The thread ApplicationServerThread-96 is waiting on the object oracle.jbo.JboSyncLock id=460e0338 locked by the thread ApplicationServerThread55. Thread ApplicationServerThread-55 is waiting on database. It is clear that the application (framework, OC4J) code introduces a serialization. One thread is waiting on the other which is reading information from the database. Provide discovered information to the development. In most cases this is the coding problem and it has to be corrected. Otherwise the development should make a comment why this serialization happened. The “-XX:+JavaMonitorsInStackTrace” parameter is extremely useful for OC4J application troubleshooting. According to author’s experience this parameter does not add significant overhead to the OC4J tier and can be used in production environments.

Java DEADLOCK problem JVM reflects information about Java deadlocks in Java “Full thread dump” if any detected. However JVM is unable to solve this kind of problems as it happens in case of Oracle Database deadlocks. OC4J process has to be restarted in case of Java deadlock. Information about the existing deadlock reflected at the end of Java “Full thread dump”.

Full thread dump: "ApplicationServerThread-126" prio=1 tid=0x881f668 nid=0x5d59 waiting for monitor entry [0xa19ff000..0xa19ff89c] ... ... FOUND A JAVA LEVEL DEADLOCK: ---------------------------"ApplicationServerThread-58": waiting to lock monitor 0x80b9804 (object 0x4595b408, a Java.lang.Object), which is locked by "ApplicationServerThread-51" "ApplicationServerThread-51": waiting to lock monitor 0x80b9564 (object 0x45955750, a com.evermind.server.http.EvermindHttpSession), which is locked by "ApplicationServerThread-58" JAVA STACK INFORMATION FOR THREADS LISTED ABOVE: -----------------------------------------------Java Stack for "ApplicationServerThread-58": ========== at oracle.jbo.http.HttpContainer.setValue(HttpContainer.Java:352) - waiting to lock <4595b408> (a Java.lang.Object)

As you can note there is the complete information about locked object and blocked threads. Provide this information to the development and restart OC4J process. You have to monitor regularly the availability of OC4J process and in case of Java deadlock to restart OC4J. This is not happening automatically by default. Author successfully used a shell scripting for this purposes.

Log files There are a dozen of log files within the standard OC4J installation. Author would like to pay your attention to two of them.

OC4J output file We have discussed this file in the context of Java “Full thread dump” (see. “Locating OC4J output file” to get location of the file). Apart from Java dump, a lot of information is going to this file, for example: • TaskManager11 process output – if you are switched on TaskManager’ debug parameter you can monitor whether the process is functioning. The standard report about the thread pool status appears in output file at the end of each TaskManager working cycle: 1108429632732:ThrP[100] ThrQ[0] 1108429632732:CxP[100] CxQ[0] - WebC[0] RmiC[0] 1108429633742:ThrP[100] ThrQ[0] 1108429633742:CxP[100] CxQ[0] - WebC[0] RmiC[0] 1108429634752:ThrP[100] ThrQ[0] 1108429634752:CxP[100] CxQ[0] - WebC[0] RmiC[0]

If output doesn’t appear in the output file it indicate the problem in the TaskManager process work. •

OC4J process debug information – if you set any OC4J debug switches (e.g. Dajp.io.debug=true) the debug information appears in OC4J output file. • Standard java application output – if application developer uses System.out.println function in the application code the output appears in OC4J output file. As you can see there is a lot of useful information in this file. Unfortunately there is not any clear indication of the event time in the file. Author finds useful to add

TaskManager process discussed in more details in “OC4J Internals” part of this paper

regularly a timestamp to the file. You can achieve this by adding line below 12to the UNIX user crontab: * * * * * echo "Timestamp: `date`" >> /u01/app/oracle/product/J2EE01/opmn/logs/PANDY.default_island.1

OC4J process output file is one of the best sources of the troubleshooting information.

OC4J default-web-access.log This file is an analogue of Apache access_log file. The information of each request processed by OC4J process is reflected in the file. This information can be extremely useful to track which OC4J process have processed the request in case you are using configuration with many OC4J processes acting as single OC4J instance13.The example of file location is seen below: OH/ j2ee/PANDY/log/PANDY_default_island_1/default-web-access.log

OS Monitoring commands Please note that multithreading implementation differs and strongly depends on used OS. Monitoring commands used in this section are relative to Linux OS. Look for equal functionality commands for your OS.

As it was mentioned before each source of OC4J troubleshooting information does not provide a complete view of the problem. You have to use all the appropriate methods to get sufficient information to troubleshoot the problem. By author’s experience the most useful sources of OC4J process runtime information are Java “Full thread dump” and OS commands. Use “nid=0x3ef2” – thread’s id provided in Java dump to connect Java dump information to the information gathered by OS commands. When the problem occurs you can go from Java dump to OS commands and to reverse the direction. For example if during OC4J unavailability you observe the high utilization of CPU (using vmstat, iostat, uptime commands), you can use top or ps -eo pcpu,pid,args | sort -k1n | tail -10 OS commands to detect top CPU consumers. Note pid-s of the processes and convert them to the hexadecimal values. Make a couple of Java “Full thread dump” and navigate to the threads you have identified by OS command. Note the information about the status of the thread (in most cases the state of top CPU consumers’ threads will be runnable), java stack (provides the information about the executed application code) and name of the thread. Provide information you gathered to the developers for further investigation. In other cases the reverse investigation path is more appropriate. Making Java “Full thread dump” you observe that many Java threads are waiting in the waiting for monitor entry state for the object locked by the tread in the runnable state. Use OS command to identify if thread is consuming resources or is waiting to receive the information from TCP/IP socket. For this purpose you can use ps command as shown below: $ while [ 1 ] ; do ps -p 13285 -o pcpu | grep -v CPU ; sleep 1 ; done 25.3 25.3 12 13

Replace the name of the output file to correct one for your environment OC4J Performance section for information about multiple OC4J configuration

25.3 $ while done 1095372 1095372 1095372

[ 1 ] ; do ps -p 19273 -o vsz,rss | grep -v VSZ ; sleep 1 ; 258244 258244 258244

In case the thread is using CPU resource you can use OS tracing commands (described in the next section) to get the additional information. Author has found the OS monitoring commands listed below as the most useful for OC4J process troubleshooting: •

System wide: vmstat, iostat, uptime, pstree

•

Top consumers of the resources top, ps -eo pcpu,pid,args | sort -k1n | tail -10

•

Particular process information while [ 1 ] ; do ps -p <pid> -o pcpu | grep -v CPU ; sleep 1 ; done while [ 1 ] ; do ps -p <pid> -o vsz,rss | grep -v VSZ ; sleep 1 ; done

OS Tracing commands Many operational systems (OS-s) have tracing utilities. They are able to attach to running process and to reflect all calls made by process to the OS. In some situations OS processes’ tracing capabilities add invaluable information for problems troubleshooting. Let us illustrate OS tracing commands’ possibilities within OC4J troubleshooting context by the example. You have noted for example that there are two processes consuming most of CPU resources for a longer time using top command: $ top -d 1 3:23pm up 7 days, 5:24, 4 1080 processes: 1076 sleeping, CPU2 states: 5.0% user, 21.0% CPU3 states: 25.0% user, 64.0% ... PID 27021 26018 9514 16047 1

USER oracle oracle oracle root root

PRI 25 15 15 15 15

users, load average: 1.36, 1.27, 1.49 3 running, 0 zombie, 1 stopped system, 0.0% nice, 72.1% idle system, 0.0% nice, 9.1% idle

NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 0 587M 587M 25096 R 65.7 15.2 244:22 java 0 11228 10M 8184 S 36.0 0.2 165:02 httpd 0 2232 2228 836 R 8.1 0.0 0:00 top 0 1172 1172 852 S 6.3 0.0 15:11 cmahostd 0 524 524 452 S 0.0 0.0 0:04 init

One of the processes is the Apache child process, the second – java thread. From Java dump you can see that the thread with id 27021 is in the runnable state. Java stack indicates that this thread is sending AJP packages: "ApplicationServerThread-31" prio=1 tid=0x69cd3c50 nid=0x698d runnable [0xba7ff000..0xba7ff89c] at java.net.SocketOutputStream.socketWrite(Native Method) at java.net.SocketOutputStream.write(SocketOutputStream.java:60) at com.evermind.server.http.AJPBodyInputStream.refillReadBuffer(AJPBodyInputStream.java:130) at com.evermind.server.http.AJPBodyInputStream.read(AJPBodyInputStream.java:102) at com.evermind.server.http.EvermindServletInputStream.read(EvermindServletInputStream.java:193) at com.evermind.server.http.EvermindHttpServletRequest.parsePostData(EvermindHttpServletRequest.java:2103) at com.evermind.server.http.EvermindHttpServletRequest.parseParameters(EvermindHttpServletRequest.java:2047) at com.evermind.server.http.EvermindHttpServletRequest.getParameter(EvermindHttpServletRequest.java:2133) at javax.servlet.ServletRequestWrapper.getParameter(ServletRequestWrapper.java:203) at _nsk._nsk001._SP0301F__Save._jspService(_SP0301F__Save.java:68) at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:56) at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:317) at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:476) at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:390) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65)

at at at at at at at at

oracle.security.jazn.oc4j.JAZNFilter.doFilter(JAZNFilter.java:283) com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:566) com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:306) com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:767) com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:166) com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:72) EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:804) java.lang.Thread.run(Thread.java:479)

By tracing the java thread we got the following information: $ strace –s1000 -p 27021 --- SIGSTOP (Stopped (signal)) --read(104, "\0224\0\2\0\0", 4096) send(104, "A", 1, 0) send(104, "B", 1, 0) send(104, "\0", 1, 0) send(104, "\3", 1, 0) send(104, "\6", 1, 0) send(104, "\37", 1, 0) send(104, "\372", 1, 0) read(104, "\0224\0\2\0\0", 4096) send(104, "A", 1, 0) send(104, "B", 1, 0) send(104, "\0", 1, 0) send(104, "\3", 1, 0) send(104, "\6", 1, 0) send(104, "\37", 1, 0) send(104, "\372", 1, 0) read(104, "\0224\0\2\0\0", 4096) send(104, "A", 1, 0) send(104, "B", 1, 0) send(104, "\0", 1, 0) send(104, "\3", 1, 0) send(104, "\6", 1, 0) send(104, "\37", 1, 0) send(104, "\372", 1, 0) read(104, "\0224\0\2\0\0", 4096) ...

= = = = = = = = = = = = = = = = = = = = = = = = =

6 1 1 1 1 1 1 1 6 1 1 1 1 1 1 1 6 1 1 1 1 1 1 1 6

The process is sending continuously the string “AB\0\3\6\37\372” and is receiving “\0224\0\2\0\0”. The apache process’s tracing gives us the following results: [root@ap1 root]# strace –s1000 -p 26018 recv(30, "\3", 1, 0) = 1 recv(30, "\6\37\372", 3, 0) = 3 send(30, "\0224\0\2\0\0", 6, 0) = 6 recv(30, "AB\0", 4, 0) = 3 recv(30, "\3", 1, 0) = 1 recv(30, "\6\37\372", 3, 0) = 3 send(30, "\0224\0\2\0\0", 6, 0) = 6 recv(30, "AB\0", 4, 0) = 3 recv(30, "\3", 1, 0) = 1 recv(30, "\6\37\372", 3, 0) = 3 send(30, "\0224\0\2\0\0", 6, 0) = 6 recv(30, "AB\0\3", 4, 0) = 4 recv(30, "\6\37\372", 3, 0) = 3 send(30, "\0224\0\2\0\0", 6, 0) = 6 recv(30, "AB\0\3", 4, 0) = 4 recv(30, "\6\37\372", 3, 0) = 3

Obviously two processes send the same packages to each other in the infinitive loop. This explains why these two processes are consuming a lot of CPU resources.

The information gathered using OS tracing capabilities was invaluable for the support organization. After this information was provided to the development, the problem in the code was identified and corrected. Please note that OS tracing functionality highly depends on operational system you use. The strace utility used in the example is Linux specific. Most of the OS-s have tracing utilities and they can be very useful during OC4J troubleshooting scenarios.

Apache side information Apache Status page There is a build-in status page in the Apache server. During unavailability cases this is the additional source of information. You navigate to the “status page” using URL provided below14: http://<server name>:<apache port>/server-status?refresh=15 Live example: http://ap10.zont.lv:7780/server-status?refresh=15

Apache “status page” provides you with apache side information. There is the information about Apache child processes such as: active processes count, processes waiting for OC4J, available Apache child processes etc. This information helps you to evaluate the situation correctly.

Apache log files Apache log files are located in the $OH/Apache/Apache/logs directory. Author has found that the information provided in the Apache log files can be misleading or interpreted incorrectly in the context of OC4J troubleshooting. Let us illustrate it by example. Often in the OC4J environment you can see messages about 500 errors in the access_log Apache log file: 192.168.2.12 - - [01/Feb/2005:16:43:34 +0200] "GET /PANDY/NSR/NSR003/NT0291F_Browse.jsp HTTP/1.1" 500 0

In the erorr_log file there are the sets of error messages: [Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0080: After calling Apache's ap_bwrite(), got an return value: -1 and the client could have aborted in the middle of receiving the chunk data. [Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0058: Failed to handle response chunk data passed in from oc4j via ajp13. [Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0035: After hdndling ajp13 response message, got an invalid value: -2. [Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0121: Failed to service request with network worker: PANDY_15 and it is not recoverable. [Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0013: Failed to call destination: PANDY's service() to service the request.

These messages mean that user of the application had pressed Stop Internet Browser button. Connection has been interrupted. After the response receiving from OC4J Apache child process was unable to send it to the Browser. As was demonstrated above Apache log files’ information can be misleading and error messages in apache log files don’t always indicate the problem.

Please note that you have to allow access to the page by editing OH/Apacha/Apache/conf/httpd.conf file.

Java code decompilation Unlike units written in other languages Java classes can be decompiled easily. There are many utilities available for this purpose15. As mentioned above OC4J (oc4j.jar) and other related components (JDBC, BC4J, etc.) are written in Java and can be decompiled. For troubleshooting purposes this adds information for you investigation. You can unzip the related archive files and decompile class files. Author has found useful to decompile java components listed below: OC4J OH/j2ee/home/oc4j.jar JDBC OH/jdbc/lib/classes12.zip OH/jdbc/lib/classes12dms.jar DCM OH/dcm/lib/dcm.jar OH/dcm/lib/oc4j_deploy_tools.jar OH/dcm/lib/oc4j_remote_deploy.jar DMS OH/lib/dms.jar BC4J OH/BC4J/redist/bc4j.ear OH/BC4J/jlib/* OH/BC4J/lib/*

Debugging Switches OC4J debugging switches documented well in the Oracle AS 10g Documentation set Containers for J2EE User's Guide 10g Release 2 (10.1.2) Part No. B14011-01 B Additional Information B.8 OC4J Command-Line Options and System Properties If your OC4J environment is on the earlier version you can still use the book mentioned above. Author has found that this is the first release of OC4J documentation where the debugging switches described relatively well. You set OC4J debugging switches in the OH/opmn/conf/opmn.xml file (java-option section). There is the example of OC4J output file with AJP debugging switch turned on (-Dajp.io.debug=true) shown below. 15

One of these utilities is available from http://www.kpdus.com/jad.html

Output file PANDA.default_island.1 1107387907360:ThrP[50] ThrQ[0] 1107387907360:CxP[20] CxQ[0] - WebC[0] RmiC[0] AJP: Input request length is: 1 AJP: AJPOutputStream::write() of 1 AJP: AJPOutputStream::writeChunk(): 1 1107387908370:ThrP[50] ThrQ[0] 1107387908370:CxP[20] CxQ[0] - WebC[0] RmiC[0]

Each switch is useful in different situation. Some of available useful categories are: ajp, http, jms, rmi, jdbc. Please do not forget to test each change of the configuration in the test environment before to implement it in the production.

Documentation, External searches and Metalink Documentation Author has found that 10g (10.1.2) Documentation set describes better OC4J related questions comparing to previous realises of Documentation (9.0.2/3/4). The concepts described in 10g documentation are valid for the earlier releases as well. You will get a better understanding of OC4J reading 10g documentation set. Author recommends to review books listed below for better understanding of OC4J related questions and troubleshooting possibilities: Oracle AS Documentation Library 10g Release 2 (10.1.2) Containers for J2EE User's Guide (144) Containers for J2EE Standalone User's Guide (134) Server Administrator's Guide (542) OPMN Administrator's Guide (158) DCM Administrator's Guide (116) Server Performance Guide (234) Oracle AS Documentation Library 10g (9.0.4) Oracle® Application Server 10g Best Practices (148) In case you are using BC4J based application please note that most of information regarding BC4J usage is located in JDeveloper Online Help system.

External searches Unfortunately there is no much information about OC4J. Sometimes you can get information you are interested in using www.google.com search capabilities. http://otn.oracle.com has a couple of forums on OC4J related questions; unfortunately the searching capabilities are very limited there. Try to ask question you are interested in and probably a person from Oracle will answer you. http://metalink.oracle.com does not have OC4J related forums and available information is rather poor. There are no many technical notes related to OC4J technology and OC4J troubleshooting. Author however has found very useful to

review Metalink Patches section of the Metalink to get the solution for some critical scalability problems.

OC4J Performance In addition to the troubleshooting methods described in “OC4J troubleshooting” part this section describes author’s findings in the context of OC4J performance tuning and available performance monitoring capabilities as well as other author’s recommendations.

Our Findings Java Workers OC4J allows you to set a minimal number of ApplicationServerThread (Workers16), as well as the maximum number of Workers and the size of requests’ queue. In the ideal situation when workers do not block each other and the database responds quickly to any request sent by Worker threads, the initial number of workers can be set to the minimal value (20) or even to be unset17. Author however has found that the setting of the minimal number to the highest value improves OC4J instance availability, scalability and overall response time. If there is a blocking situation in your environment and there is no available worker thread, OC4J will put the requests of each user in a queue. OC4J will spawn the additional Worker thread only when queue achieves the limit set in the configuration file. This behavior can significantly limit OC4J process availability and performance. Set the Worker thread’s parameters in the server.xml file (see. example below). OH/j2ee/<comp_name>/config/server.xml <application-server ...> ... <global-thread-pool min="100" max="300" queue="30" keepAlive="-1" cx-min="100" cx-max="300" cx-queue="30" cx-keepAlive="-1" debug="true"/> ... </application-server>

Setting the min parameter to the high value you make OC4J free from resources’ intensive procedure of creating the new Workers, as well as from queue managing routine. Reader can argue that the high number of unused threads adds the additional overhead and utilization of CPU resources. Author however did not note the significant overhead while setting the parameter in the production environments.

OC4J Processes It is possible to startup the multiple OC4J processes, which act as one OC4J instance. mod_oc4j (Apache module) will spread users’ requests across all the available OC4J processes. Even if OC4J instance is located on a single host you can earn a significant improvement in terms of system availability and performance by distributing the load through multiple OC4J instances. Set numProcs parameter in opmn.xml file for particular instance. There are three processes for instance PANDY in the following example. OH/opmn/conf/opmn.xml 16

Workers – are the threads which process users’ requests and execute the application code. For the additional information about this type of threads see “OC4J Internals” 17 If you do not set the initial number of Workers OC4J initializes 1 Worker thread at the startup

Please note that OPMN will start three dedicated OC4J processes. Each of them will utilize its own memory areas. In the example above all the PANDY instance OC4J processes will utilize 1.5 GB of RAM. Ensure that you have the sufficient memory resources for all instances. Please note that each process acts independently, each process will have separate DMS and BC4J statistics, as well as own AJP/RMI/JMS ports. The mod_oc4j however will send usersâ&#x20AC;&#x2122; requests to one particular OC4J process. Routing information for each user session is stored in the JSESSIONID coke18.

Memory Author has found reasonable to set the minimal size of memory available to the single OC4J process equal to the maximum size. However this is not a strict recommendation. You set memory usage parameters in opmn.xml file: OH/opmn/conf/opmn.xml <oc4j â&#x20AC;Ś <java-option value="-server -Xms512M -Xmx512M"/> </oc4j>

Setting min=max JVM does not need memory to be allocated dynamically. Each OC4J process reserves memory during startup. You have to monitor regularly the usage of memory within OC4J process. For this purpose you can set -verbose:gc java parameter, as well as dmstool functionality. DMS utility allows you to monitor memory usage any time: $ dmstool -table JVM | grep 3001 | grep Memory /ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.value /ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.minValue /ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.maxValue /ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.value /ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.minValue /ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.maxValue ap1.voava.lv*J2EE01-/u01/app/oracle/product $

495796 kbytes 495796.0 kbytes 522468.0 kbytes 524032 kbytes 524032.0 kbytes 524032.0 kbytes

The single snapshot of memory metrics does not add any significant value. It might be more effective to collect metrics regularly.

You can observe cookie by entering javascript:alert(document.cookie); string in the Internet Browser address bar after you logged to OC4J application.

The above example reflects the typical behavior of OC4J JVM. Garbage Collector (GC) frees the significant memory areas periodically. There is a lot of information regarding the tuning of GC. For example you may find the following links useful: http://java.sun.com/docs/hotspot/gc/ http://developers.sun.com/techtopics/mobility/midp/articles/garbagecollection2/ http://www.cons.org/cmucl/doc/gc-tuning.html

Grep OC4J log files for out of memory errors. If you allocated a significant amount of memory it is possible that there are the memory leaks in the application code. Ask the application support or the development organizations for memory allocation recommendations. In case of memory leaks Java Profilers19 such as JProfiler from Quest Software or others can be used to locate and to find root of the problem. Be aware that Java Profilers add significant overhead and are not usable normally in the production environments.

Stresstesting Often a need for the stresstesting is underestimated. However, according to authorâ&#x20AC;&#x2122;s experience the stresstesting is one of the best methods to tune OC4J application performance. Weak areas can be identified easily. Author considers using the automatic stresstesting tools. This will allow you to repeat the testing for several times and to estimate the code correction results. It is almost impossible to test all the functionality and dynamic behavior of system users. Include the most used forms and functionality in the stresstesting scenarios. Repeat testing â&#x20AC;&#x201C; performance the problem correction cycles for several times until you get the appropriate results. By implementing the stresstesting process you can avoid a lot of problems in future.

The profiling with Oracle JDeveloper is possible on Win32 platform only.

Response Time Monitoring by DMS

Apache OHS [c]

Send

Request B

OC4J mod_plsql handle Handle B

JVM [java]

ApplicationServerThread Read Resolve

Process B JSP Processing Servlet Processing

mod_oc4j handle

Process E Handle E Handle B Handle E

Request E

Recive

mod_perl mod_php ...

DMS Sensors

DMS-client DMS.JAR (java) statistics reflecting

The figure above reflects the example of single userâ&#x20AC;&#x2122;s request processing. Apache as well as OC4J and other Oracle AS components have the build-in DMS (Dynamic Monitoring Services) sensors, which reflect different performance metrics. You can monitor this statistics using dmstool.

Response Time Breakdown Example $ dmstool -table ohs_server | grep time .../Apache/connection.time 48123608767 usecs .../Apache/request.time 15916131942 usecs .../Apache/handle.time 15913988395 usecs $ $ dmstool -table ohs_module | grep time | sort -k2n | tail -2 .../Apache/Modules/mod_dms.c/handle.time 8081655 usecs .../Apache/Modules/mod_oc4j.c/handle.time 15905072698 usecs $ $ dmstool -table oc4j_jspExec | grep time .../oc4j/PANDY/WEBs/PANDY/JSPs/processRequest.time 17 563 793 msecs $ $ $ $ dmstool -table "oc4j_jsp(threadsafe=true)" | grep time | sort -k2n | tail -5 .../oc4j/PANDY/WEBs/PANDY/NS0285F_searchForm.jsp/service.time 1 072 570 .../oc4j/PANDY/WEBs/PANDY/D2261F_PersonsBrowse.jsp/service.time 1 407 663 .../oc4j/PANDY/WEBs/PANDY/NS0291F_Browse.jsp/service.time 2 942 655 .../oc4j/PANDY/WEBs/PANDY/D2261F_Edit.jsp/service.time 3 151 184 .../oc4j/PANDY/WEBs/PANDY/D2260F_Browse.jsp/service.time 4 847 408 $

msecs msecs msecs msecs msecs

As it is shown in the example you can track Apache response time. Unfortunately DMS do not have the jet capabilities as event 10046 within Oracle database. However they can give you the answers to the questions like: -

Which handle (Apache module) has taken the most of the response time?

How much time the JSP processing has taken?

Which JSP has taken the longest time to execute?

DMS Sensors Classification You can observe all the available DMS sensor types by entering dmstool -list -table command. This command reflects thousands of different metrics (depending on configuration). It is difficult to identify the important ones. There is the DMS metric classification below to simplify the process of identification: • Informative sensors – reflect a configuration information and some system wide statistics o ohs_child, ohs_responses, o JVM, oc4j_ear, oc4j_opmn, oc4j_task, o JDBC_Driver , JDBC_Connection • Average performance sensors – reflect response time totals by modules, applications, etc. o ohs_server , ohs_module, ohs_virtualHost, o oc4j_context, oc4j_web_module, oc4j_jspExec •

Units performance sensors – reflect response time statistics for each unit o oc4j_servlet o oc4j_jsp(threadsafe=true)

Collecting and Reflecting DMS Statistics Be aware that the DMS Sensor statistics is incremental. You have to make the periodical snapshots of statistics to get a time dimension for further analysis. Develop the interface to access statistics easily. For example you may reflect the average performance sensors graphically:

Or develop STATSPACK like interface to reflect Units performance metrics: SYS:PROD> @sp_jsp Completed Snapshots Snap Id ------530 531 532 549

Snap Started ------------------2005-02-03 12:33:26 2005-02-03 13:03:24 2005-02-03 13:33:27 2005-02-03 14:03:32

Specify the Begin and End Snapshot Ids ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Enter value for begin_snap: 532 Enter value for end_snap: 549 Saved in c:\temp\532_549.log Snap Id Snap Time ------- -----------------Begin Snap: 532 2005-02-03 13:33:27 End Snap: 549 2005-02-03 14:03:32 Elapsed: 30.08 (mins) JSP ordered by Response time JSP Name Response Execution Average (sec) Time(sec) Count Response ------------------------------------------------------------------ --------- ---------- ---------../3001/../NSK/NSK001/AP3220F_TicketImportProcessOneByOne.jsp 956.439 2 478.22 ../3002/../NSK/NSK001/AP3220F_Tickets_browse.jsp 708.967 71 9.99 ../3001/../NSK/NSK001/M2801F_Browse.jsp 533.389 74 7.21 ../3003/../NSK/NSK001/P2600F_XMLImportProcess.jsp 471.106 1 471.11 ../3003/../LRG/l001/L3000F_Browse.jsp 293.775 92 3.19 ../3002/../NSK/NSK001/AP3221F_ApnAccountingDocFilter.jsp 291.382 6 48.56 ../3002/../common/comp/CMN_DataFilterComponent.jsp 238.75 35 6.82 ../3001/../NSK/NSK001/SP0301F.jsp 216.499 32 6.77 ../3001/../SKD/d001/D2262F_Browse.jsp 208.709 7 29.82 ../3001/../NSK/NSK001/SP0301F_filter.jsp 168.957 6 28.16 ../3002/../SKD/d001/D2260F_Browse.jsp 138.453 45 3.08 ../3003/../SKD/d001/D2260F_Browse.jsp 81.773 24 3.41 ../3001/../NSK/NSK001/Z3601F_Save.jsp 80.127 30 2.67 ../3001/../common/comp/CMN_LOVcomp.jsp 73.544 52 1.41 ../3003/../NSK/NSK001/SP0301F.jsp 72.548 29 2.5 ../3001/../SKD/d001/D2260F_Browse.jsp 71.642 62 1.16 ../3002/../NSK/NSK001/AP3221F_Save.jsp 68.509 91 .75 ../3001/../common/comp/CMN_DataHandlerComponent.jsp 68.46 765 .09 ../3002/../common/comp/CMN_DataHandlerComponent.jsp 57.144 722 .08 SYS:PROD>

Appendix A. How to diagnose opened files, sockets, pipes by OC4J process? It can be useful for diagnostic and troubleshooting purposes to see all files opened by the process. For this purposes you can use lsof like utility20. The lsof utility lists all files opened by the process. There is an example of lsof output for OC4J process below (see Figure 3 lsof utility output example). Using mentioned utility you can see easily all the log files, sockets, *.JAR files opened by OC4J process. Itâ&#x20AC;&#x2122;s interesting to see for example that OC4J process uses different *.JAR files from directories listed below: /u01/app/oracle/product/J2EE01/lib/dms.jar /u01/app/oracle/product/J2EE01/opmn/lib/ons.jar /u01/app/oracle/product/J2EE01/j2ee/home/oc4j.jar /u01/app/oracle/product/J2EE01/j2ee/home/lib/http_client.jar /u01/app/oracle/product/J2EE01/config/Queries.jar ...

lsof utility is available on Linux and Sun Solaris platforms. Look for similar utility on your platform.

Figure 3 lsof utility output example COMM PI USER FD TYPE AND D java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java

8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010

oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle

18u 13u rtd 0r mem mem mem mem mem mem mem mem mem mem mem cwd mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem 7w 9w 35r 49r 32r 54r 28r 29r 38r 37r 43r 30r 36r 34r 52r 45r 10w 6w 12w 15w 5w 11w

Ipv4 -1 Ipv4 -1 DIR CHR REG REG REG REG REG REG REG REG REG REG REG DIR REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG

DEVIC E 79444636 79444643 104,7 1,3 104,7 104,7 104,7 104,7 104,7 104,7 104,7 104,7 104,7 104,7 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8

SIZE

1024 466557 5774329 65509 616450 451085 73929 257535 351452 561991 263234 120257 4096 1431995 37265 126451 938119 45386 726013 515920 17978 58142 206289 12822 228847 104195 1070096 28404 6727 424080 27724 98496 8674 280984 605174 53389 4783072 1114323 6690838 1438 1422 1433 1339 1359 1386 1418 3761 1496 1464 6031 1387 7517 6657 55981920 133330 42600 85101 148705 10

NODE

NAME

TCP TCP 2 9511 60577 60490 60494 60496 60499 60512 60515 60523 60526 60528 2998503 3571783 3571807 3571808 3571810 3571789 2769040 2769036 2769054 2769032 2769033 2769041 2769029 2769037 2769030 2769038 2769031 2769034 2769039 2769044 2769047 2769048 2769049 2769001 2769051 3571813 5030160 6340902 2184291 2184306 2184453 2184505 2184528 2184557 2184596 2184998 2185078 2185147 2185426 2186115 2186174 2186205 5095446 5095445 5095448 5095449 5095444 6865677

*:3001 (LISTEN) *:3201 (LISTEN) / /dev/null /lib/ld-2.2.4.so /lib/libc-2.2.4.so /lib/libdl-2.2.4.so /lib/libm-2.2.4.so /lib/libnsl-2.2.4.so /lib/libnss_dns-2.2.4.so /lib/libnss_files-2.2.4.so /lib/libnss_nisplus-2.2.4.so /lib/libpthread-0.9.so /lib/libresolv-2.2.4.so /u01/app/oracle/product/J2EE01/config/Queries.jar /u01/app/oracle/product/J2EE01/j2ee/home /u01/app/oracle/product/J2EE01/j2ee/home/iiop.jar /u01/app/oracle/product/J2EE01/j2ee/home/iiop_gen_bin.jar /u01/app/oracle/product/J2EE01/j2ee/home/jaxb-rt-1.0-ea.jar /u01/app/oracle/product/J2EE01/j2ee/home/jazn.jar /u01/app/oracle/product/J2EE01/j2ee/home/lib/activation.jar /u01/app/oracle/product/J2EE01/j2ee/home/lib/aqapi.jar /u01/app/oracle/product/J2EE01/j2ee/home/lib/bcel.jar /u01/app/oracle/product/J2EE01/j2ee/home/lib/connector.jar /u01/app/oracle/product/J2EE01/j2ee/home/lib/cos.jar /u01/app/oracle/product/J2EE01/j2ee/home/lib/crimson.jar /u01/app/oracle/product/J2EE01/j2ee/home/lib/ejb.jar /u01/app/oracle/product/J2EE01/j2ee/home/lib/http_client.jar /u01/app/oracle/product/J2EE01/j2ee/home/lib/jaas.jar /u01/app/oracle/product/J2EE01/j2ee/home/lib/jasper.zip /u01/app/oracle/product/J2EE01/j2ee/home/lib/jaxp.jar /u01/app/oracle/product/J2EE01/j2ee/home/lib/jdbc.jar /u01/app/oracle/product/J2EE01/j2ee/home/lib/jem.jar /u01/app/oracle/product/J2EE01/j2ee/home/lib/jms.jar /u01/app/oracle/product/J2EE01/j2ee/home/lib/jndi.jar /u01/app/oracle/product/J2EE01/j2ee/home/lib/jta.jar /u01/app/oracle/product/J2EE01/j2ee/home/lib/mail.jar /u01/app/oracle/product/J2EE01/j2ee/home/lib/ojsp.jar /u01/app/oracle/product/J2EE01/j2ee/home/lib/servlet.jar /u01/app/oracle/product/J2EE01/j2ee/home/oc4j.jar /u01/app/oracle/product/J2EE01/j2ee/PANDA/application-deployments/BC4J/PANDA_default_island_1/application.log /u01/app/oracle/product/J2EE01/j2ee/PANDA/application-deployments/PANDA/PANDA_default_island_1/application.log /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/entity/ApnNskAdDiagnosis_AS.xml /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/entity/ApnNskForward_AS.xml /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/entity/NmpNskAdClassify_AS.xml /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/entity/NskAdDiagnosis_AS.xml /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/entity/NskAdPatientGroups_AS.xm /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/entity/NskCadAdsFkAssoc.xml /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/entity/NskHpsAdsFkAssoc.xml /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/nsk001/P2601F_NmpAdAttributes_V /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/nsk001/SP0301F_NskAccountingDoc /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/nsk001/SP0301F_NskXmlDocErrors_ /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/nsk002/P2631F_NskLcSubcriteria_ /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/skd/d001/D2262F_Persons_Det5_VL.xml /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/skd/entity/SkdPersonDiagnoses_EO.xm /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/skd/entity/SkdVcurrentPvaDoctors.xm /u01/app/oracle/product/J2EE01/j2ee/PANDA/log/PANDA_default_island_1/default-web-access.log /u01/app/oracle/product/J2EE01/j2ee/PANDA/log/PANDA_default_island_1/global-application.log /u01/app/oracle/product/J2EE01/j2ee/PANDA/log/PANDA_default_island_1/jms.log /u01/app/oracle/product/J2EE01/j2ee/PANDA/log/PANDA_default_island_1/rmi.log /u01/app/oracle/product/J2EE01/j2ee/PANDA/log/PANDA_default_island_1/server.log /u01/app/oracle/product/J2EE01/j2ee/PANDA/persistence/PANDA_default_island_1/transaction.state

java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java java

mem mem mem txt mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem 1w 2w 8w mem mem mem mem mem 22w mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem mem 53u 50u 16u 24u 26u 41u 33u 39u 42u 23u

REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG REG IPv4 IPv4 IPv4 IPv4 IPv4 IPv4 IPv4 IPv4 IPv4 IPv4

-1 -1 -1 -1 -1 -1 -1

104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,8 104,5 104,5 104,5 104,5 104,5 104,5 104,5 104,5 104,5 104,5 104,5 104,5 104,5 104,5 104,5 104,5 104,5 104,5 104,5 104,5 988260755 9403290 79444641 33559729 982099578 981641728 30681729 20319458 20230199 46160142

400813 1227663 1846926 27090 3228357 3968300 4323150 221718 214414 45805 81155 100867 60924 13887946 86267 4723739 47977 19055 97158 50921 162059 246743 23443 50413 260225 463863 658404 53464 23185778 23185778 23185778 297060 419100 3717072 142704 123656 7154 1144432 160 21499 173408 370 28 57 291 82 59 39 64 2456 82080 31544 1009336 51632 26788 306428 17712

2293843 5783844 5783845 3998180 6357412 2441654 6357389 6357381 6357388 6357385 6357380 6357382 2441652 6357411 6357399 3522887 3719344 3719459 3719348 3719462 3719351 3719457 3719364 3719352 5783839 5783772 5783805 6832352 6144409 6144409 6144409 819453 6471908 6471909 3276954 3276958 2999584 98829 427556 213158 98491 131090 263082 196715 262181 427555 98484 393342 426055 132545 361727 361731 361733 361741 361753 361759 361761 TCP TCP TCP TCP TCP TCP TCP TCP TCP TCP

/u01/app/oracle/product/J2EE01/javacache/lib/cache.jar /u01/app/oracle/product/J2EE01/jdbc/lib/classes12dms.jar /u01/app/oracle/product/J2EE01/jdbc/lib/nls_charset12.jar /u01/app/oracle/product/J2EE01/jdk/bin/i386/native_threads/java /u01/app/oracle/product/J2EE01/jdk/jre/lib/i18n.jar /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/client/libjvm.so /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/libawt.so /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/libjava.so /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/libmlib_image.so /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/libnet.so /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/libverify.so /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/libzip.so /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/native_threads/libhpi.so /u01/app/oracle/product/J2EE01/jdk/jre/lib/rt.jar /u01/app/oracle/product/J2EE01/jdk/jre/lib/sunrsasign.jar /u01/app/oracle/product/J2EE01/jdk/lib/tools.jar /u01/app/oracle/product/J2EE01/jlib/infratool.jar /u01/app/oracle/product/J2EE01/jlib/javax-ssl-1_1.jar /u01/app/oracle/product/J2EE01/jlib/jndi.jar /u01/app/oracle/product/J2EE01/jlib/jssl-1_1.jar /u01/app/oracle/product/J2EE01/jlib/ldapjclnt9.jar /u01/app/oracle/product/J2EE01/jlib/netcfg.jar /u01/app/oracle/product/J2EE01/jlib/ojmisc.jar /u01/app/oracle/product/J2EE01/jlib/repository.jar /u01/app/oracle/product/J2EE01/lib/dms.jar /u01/app/oracle/product/J2EE01/lib/jsse.jar /u01/app/oracle/product/J2EE01/lib/xmlparserv2.jar /u01/app/oracle/product/J2EE01/opmn/lib/ons.jar /u01/app/oracle/product/J2EE01/opmn/logs/PANDA.default_island.1 /u01/app/oracle/product/J2EE01/opmn/logs/PANDA.default_island.1 /u01/app/oracle/product/J2EE01/opmn/logs/PANDA.default_island.1 /u01/app/oracle/product/J2EE01/soap/lib/soap.jar /u01/app/oracle/product/J2EE01/sqlj/lib/runtime12ee.jar /u01/app/oracle/product/J2EE01/sqlj/lib/translator.jar /u01/app/oracle/product/J2EE01/webservices/lib/wsdl.jar /u01/app/oracle/product/J2EE01/webservices/lib/wsserver.jar /u01/app/PANDA/logs/panda-2005-01-30-08-28-14-958.log /usr/lib/libstdc++-2-libc6.1-1-2.9.0.so /usr/lib/locale/en_US/LC_ADDRESS /usr/lib/locale/en_US/LC_COLLATE /usr/lib/locale/en_US/LC_CTYPE /usr/lib/locale/en_US/LC_IDENTIFICATION /usr/lib/locale/en_US/LC_MEASUREMENT /usr/lib/locale/en_US/LC_MESSAGES/SYS_LC_MESSAGES /usr/lib/locale/en_US/LC_MONETARY /usr/lib/locale/en_US/LC_NAME /usr/lib/locale/en_US/LC_NUMERIC /usr/lib/locale/en_US/LC_PAPER /usr/lib/locale/en_US/LC_TELEPHONE /usr/lib/locale/en_US/LC_TIME /usr/X11R6/lib/libICE.so.6.3 /usr/X11R6/lib/libSM.so.6.0 /usr/X11R6/lib/libX11.so.6.2 /usr/X11R6/lib/libXext.so.6.4 /usr/X11R6/lib/libXp.so.6.2 /usr/X11R6/lib/libXt.so.6.0 /usr/X11R6/lib/libXtst.so.6.1 ap1.voava.lv:28475->prod2.vovaa.lv:1526 (ESTABLISHED) ap1.voava.lv:30001->prod2.vovaa.lv:1526 (ESTABLISHED) ap1.voava.lv:3101 (LISTEN) ap1.voava.lv:31435->prod2.vovaa.lv:1526 (ESTABLISHED) ap1.voava.lv:38392->prod2.vovaa.lv:1526 (ESTABLISHED) ap1.voava.lv:39100->prod2.vovaa.lv:1526 (ESTABLISHED) ap1.voava.lv:42991->prod2.vovaa.lv:1526 (ESTABLISHED) ap1.voava.lv:45951->prod2.vovaa.lv:1526 (ESTABLISHED) ap1.voava.lv:46351->prod2.vovaa.lv:1526 (ESTABLISHED) ap1.voava.lv:46866->prod2.vovaa.lv:1526 (ESTABLISHED)

java java java java java java java java java java java java java

8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010 8010

oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle

44u 27u 40u 31u 46u 25u 21u 48u 14u 17u 19u 3r 4w

IPv4 IPv4 IPv4 IPv4 IPv4 IPv4 IPv4 IPv4 sock IPv4 IPv4 FIFO FIFO

-1 -1 -1 -1 -1 -1 -1 -1

19680489 975439727 329717 19232844 999783949 42714920 39574270 16842834 0,0 79444637 79444635 0,0 0,0

TCP TCP TCP TCP TCP TCP TCP TCP 3215522651 TCP TCP 3215522636 3215522636

ap1.voava.lv:48228->prod2.vovaa.lv:1526 (ESTABLISHED) ap1.voava.lv:48625->prod2.vovaa.lv:1526 (ESTABLISHED) ap1.voava.lv:49006->prod2.vovaa.lv:1526 (ESTABLISHED) ap1.voava.lv:50077->prod2.vovaa.lv:1526 (ESTABLISHED) ap1.voava.lv:50286->prod2.vovaa.lv:1526 (ESTABLISHED) ap1.voava.lv:52557->prod2.vovaa.lv:1526 (ESTABLISHED) ap1.voava.lv:59195->prod2.vovaa.lv:1526 (ESTABLISHED) ap1.voava.lv:59222->prod2.vovaa.lv:1526 (ESTABLISHED) can't identify protocol localhost:3101 (LISTEN) localhost:64214->localhost:7100 (ESTABLISHED) pipe pipe