Performance Tuning and Troubleshooting for Oracle OC4J
Jurijs Velikanovs 7+ years Oracle DBA, OCP 7/8/8i/9i, OCA 9iAS R2
Topics: z z z z z z
Introduction Usage considerations OC4J Internals OC4J Troubleshooting OC4J Performance tuning BC4J
Introduction
Who am I? And what are my responsibilities? z
Who am I? I
am a DBA team leader I’m neither a Java fan nor a Java Developer nor a Java Guru My team has spent a lot of time researching and struggling to stabilize the production environments based on OC4J z
Responsibilities To
support production and development environments To ensure production availability To troubleshoot production system To provide all the necessary information to the developers or support organizations to solve a problem
How were we involved? z
z
z
Local Oracle Marketing team has pushed customers to demand the use of OC4J in their tenders’ conditions In 2002 the company management decided to switch the whole Development from Oracle Forms to OC4J/BC4J/JDeveloper technology We had 100
forms developers without any Java experience A few Java fans Oracle 9iAS R2 (9.0.2), later 9.0.3.1 In 2003 the first phase of the first application was deployed
Example of the Applications we are working with
z
Healthcare financial system Enterprise
application 2+ years in the production and still under development Oracle 9iAS 9.0.3.1 OC4J, Oracle 9.2.0.5 DB JSP = 1448 files, CLASS = 3283 files 965 tables, 593 views, 251 types, 220 packages, DB Size 50 GB, REDO 2-8 GB Daily 800 heavy users per day z
10x5 hour availability
Why is it so important? z
OC4J is the present strategic direction for Oracle JDeveloper,
z
Forms, Reports, Discoverer, OEM, etc.
Difficult to troubleshoot Complex
technology Many components Java on Java
Why is it so important? z
OC4J is a strategic direction for Oracle JDeveloper,
z
Forms, Reports, Discoverer, OEM
Difficult to troubleshoot Complex
technology Many components Java on Java z z z
There is not much information about this product Poor vendor support Many Oracle customers use it and have troubles
I will not tell you how to develop in Java … I’m not going to tell you about Web and EJB application technology VMC etc.
The
technical layer is of great interest
I’m going to speak about OC4J internals … ONS
PORT: 3001
Thread
AJP Connection Listener
OC4JMonitor Thread
new req ApplicationServerThread
Thread
ApplicationServerThread
ApplicationServerThread ApplicationServerThread
Java native
ApplicationServerThread ...
AJP 1.3
ONet
ApplicationServerThread ApplicationServerThread
Memory Structures Application objects
3101
RMI RMIServer Server
3201
Http Session List
JMSServer
PORT: 3101
PORT: 3201
...
TaskManager
TaskManager JMSServer JMSServer JMSServer JMSServer Thread Thread
JDBC Connection pool
Java native VM Thread VM Thread Signal Dispatcher Signal Dispatcher Suspend Checker Suspend ThreadChecker Thread
VM Periodic Task VMThread Periodic Task Thread Finalizer Finalizer Reference Reference Handler Handler
Usage considerations or how to avoid troubles
Oracle AS Architecture Considerations z
You may make the configuration complicated, with many components highly dependent on each other
Oracle AS Architecture Considerations BUT: z Keep it as simple as possible z At first you should be sure that the system works properly in a simple configuration z Make components as independent as possible z Don’t use the infrastructure if there is no strong need for it (i.e. Single-Signon, Discoverer, Clustering)
Oracle AS Typical Architecture DCM-daemon
OPMN
DMS-metrics DMS.JAR (java) statistics reflecting
PM, ONS (c) strat, stop, ping, notification
DCM (java) configuration management
DCM repository
opmn.xml
httpd.conf
HTTP
PORT: 80, 7780
WAN
OHS (c) Child Child Server Server Child Server Child Server Child Server Child Server Child Server Child Server Child Child Server Server Child Server Child Server Child Server Child Server Child Server Child Server Child Server Child Child Server Server Child Server Child Server Child Server Child Server Child Server Child Server Child Server Child Child Server Server Child Server Child Server Child Server Child Server Child Server Child Server Child Child Server Server
JVM JVM(java) JMSServer JMSServer JMSServer Thread JMSServer Thread Thread Thread
AJP 1.3
PORT: 3001
Apache
server.xml
OC4J
AJP AJP Connection Connection Listener Listener
ApplicationServerThread ApplicationServerThread ApplicationServerThread ApplicationServerThread ApplicationServerThread ApplicationServerThread
OC4JMonitor OC4JMonitor Thread Thread (dms)
TaskManager TaskManager TaskManager TaskManager
Java native Java native VM Thread VM Thread RMI RMIServer Server PORT: 3101
JMSServer PORT: 3201
Signal Dispatcher Signal Dispatcher Suspend Checker Suspend ThreadChecker Thread
VM Periodic Task VMThread Periodic Task Thread Finalizer Finalizer Reference Reference Handler Handler
DB
Installation Considerations z z
z z
Use the latest version of Oracle AS depending on your circumstances Use the latest versions of Oracle Forms, Reports, Discoverer, Portal. Install them in the OH separated from your custom development OC4J OH Install the latest patches for OS, JDK, OC4J, JDBC components Install the version of BC4J which was used by the Development. Don’t use the default version of BC4J/ADF/JHeadstart (See. JDeveloper install.html)
Operational Considerations z z
Keep running only the components you need Don’t startup in the production environment: Oracle
Enterprise Manager 10g Application Server Control OC4J_Demos component “home” component z z z
Don’t deploy your applications to the “home” component Create your own component and use it to deploy and run your applications Use the opmnctl, dcmctl, dmstool utilities to maintain your AS environment
Deployment Considerations z
Get a commitment from the Development to use industry standards for the Application development
Use EAR files to deploy any application changes Use Ant tool (http://ant.apache.org) to make EAR/WAR files In emergency cases use WAR files to deploy small changes to application z z z
Use undeployApplication/deployApplication combination rather than redeployApplication dcmctl command Use dcmctl saveInstance before deploying to backup current configuration Deploy BC4J.EAR to your OC4J component to keep “home” down
Versions Considerations Recommendations a dedicated test environment and verify all the changes there before implementing in production If you have the automatic stresstesting scenarios, run them after each change in the test environment Monitor the impact of changes and if the situation becomes worse rollback them
Application
Have
APP Frame Work BC4J 9.0.3.10.7 OC4J
JDBC
c.c.c.x 9.0.3.1
c.c.c.x 9.2.0.5
JDK c.c.x_x, 1.3.1_15 OS 2.4.9-e.38enterprise
Versions Considerations z
OS
z
It is important to install the latest patches from an OS vendor (Kernel, Network, etc.) Usually the patches can be rollbacked easily The change of OS main release requires more testing and efforts than patching
JDK
Install the latest versions of JDK within c.c. boundaries Use the JDK from Sun ww.JavaSoft.com To install it replace the OH/jdk directory. These changes can be rollbacked easily The change of c.c. version requires the Development involvement and more testing
Application APP Frame Work BC4J 9.0.3.10.7 OC4J
JDBC
c.c.c.x 9.0.3.1
c.c.c.x 9.2.0.5
JDK c.c.x_x, 1.3.1_15 OS 2.4.9-e.38enterprise
Versions Considerations z
JDBC
z
Install the latest version of JDBC within c.c.c. version To install it replace OH/jdbc directory. These changes can be rollbacked easily It requires the minimal functionality testing
Application APP Frame Work BC4J 9.0.3.10.7
OC4J
Can be easily upgraded within a c.c.c. version boundaries It requires the minimal functionality testing Follow the patch installation instructions (9iAS 9.0.3.1) There are 50+ one-off patches (*.class) The change of c.c.c. version requires the Development involvement and more testing
OC4J
JDBC
c.c.c.x 9.0.3.1
c.c.c.x 9.2.0.5
JDK c.c.x_x, 1.3.1_15 OS 2.4.9-e.38enterprise
Versions Considerations z
BC4J A BC4J version sets the requirements for OC4J version It can’t be changed without Development involvement even within x.x.x.x. version boundaries Often there are needs for application changes It requires a lot of testing New version can have negative impact to your application The change of c.c.c. version requires more application code changes, testing, and other efforts
Application APP Frame Work BC4J 9.0.3.10.7 OC4J
JDBC
c.c.c.x 9.0.3.1
c.c.c.x 9.2.0.5
JDK c.c.x_x, 1.3.1_15 OS 2.4.9-e.38enterprise
Versions Considerations z
How to Determine the Versions
OS: uname -r
$ uname -r 2.4.9-e.38enterprise
JRE: java –version
$ $ORACLE_HOME/jdk/bin/java -version java version "1.3.1_13" Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.1_13-b03) Java HotSpot(TM) Client VM (build 1.3.1_13-b03, mixed mode)
OC4J: java -jar oc4j.jar –version
$ $ORACLE_HOME/jdk/bin/java -jar $ORACLE_HOME/j2ee/home/oc4j.jar -version Oracle9iAS (9.0.3.1.0) Containers for J2EE (build 031214.BP1.2075)
JDBC: a program with a call to meta.getDriverVersion (location $OH/jdbc/lib/classes12dms.jar, Notes:73629.1/244074.1)
$ $ORACLE_HOME/jdk/bin/java JDBCVersion Oracle JDBC driver 9.2.0.5.0
BC4J: a program with a call to meta.getDriverVersion (location $BC4J/lib/bc4jct.jar)
$ $ORACLE_HOME/jdk/bin/java BC4JVersion 9.0.3.10.7
Other Considerations z
z z z
z
z
Have at least one senior level developer, who understands used technology well, is capable to make technical decisions and to lead the framework development, maintenance and troubleshooting Synchronize OC4J, BC4J versions with Development Use an Automatic testing tool for the application stresstesting A commitment from Oracle is desirable. If you can manage a connection to the Oracle JDeveloper development team to support your project, it will be an invaluable help during troubleshooting and decision making Have a skilled DBA for environments maintenance, monitoring and troubleshooting. The DBA provides all possible information to the development for application troubleshooting Have a monitoring framework
Before going to life
!!! Stresstest your application !!! (or ask for results)
OC4J Internals
OC4J Internals ONS
PORT: 3001
Thread
AJP Connection Listener
OC4JMonitor Thread
new req ApplicationServerThread
Thread
ApplicationServerThread
ApplicationServerThread ApplicationServerThread
Java native
ApplicationServerThread ...
AJP 1.3
ONet
ApplicationServerThread ApplicationServerThread
Memory Structures Application objects
3101
RMI RMIServer Server
3201
Http Session List
JMSServer
PORT: 3101
PORT: 3201
...
TaskManager
TaskManager JMSServer JMSServer JMSServer JMSServer Thread Thread
JDBC Connection pool
Java native VM Thread VM Thread Signal Dispatcher Signal Dispatcher Suspend Checker Suspend ThreadChecker Thread
VM Periodic Task VMThread Periodic Task Thread Finalizer Finalizer Reference Reference Handler Handler
Processes (threads) within OC4J z
Java native threads
VM Thread, VM Periodic Task Thread, Signal Dispatcher, Suspend Checker Thread, Finalizer, Reference Handler Exist in any Java based application z
OC4J Listener - threads
AJP/HttpConnectionListener, RMIServer (2), JMSServer Accepting clients requests, assigning them to the worker threads The configuration of listened ports is in opmn.xml file z
Set of processes for communicating with ONS
OC4JMonitorThread, Thread(2) These processes make connection to the OPMN immediately after the startup of OC4J They send event messages to the OPMN to be broadcasted to subscribers
Processes (threads) within OC4J z
TaskManager
Responsible for a periodic tasks executing (housekeeping)
server.http.SessionTimeoutTask (drops expired session) server.ApplicationServerTask (clear expired threads) sql.DriverManagerXADataSource …
You can set an executing frequency in server.xml file taskmanager-granularity=1000 (default 1 sec)
Use the dmstool to monitor the statistics of the process dmstool -table oc4j_task
z
ApplicationServerThreads (Global Thread Pool)
Execute users requests, run an application code, communicate with the Apache processes and the database Initial count of the processes can be set in server.xml file global-thread-pool min=“200" max="200" queue="20“
OC4J Troubleshooting
Classification of Problem Areas OC4J JVM(java) PORT: 3001
4.
AJP JMSServer JMSServer Connection Listener
Apache
New Request as sig n
ApplicationServerThread ApplicationServerThread ApplicationServerThread ApplicationServerThread ApplicationServerThread TaskManager
1.
ApplicationServerThread
2.
3.
DB
ApplicationServerThread ApplicationServerThread ApplicationServerThread
JDK OS
1. OC4J container inside (infinitive cycles, waits on other threads, deadlocks) 2. Interface to the Apache processes 3. Interface to the database 4. 500 Internal Error
Possible Approaches & Information Sources z z
Java Full thread dump Other Log
files OS Monitoring commands OS Tracing commands Apache side information Java code decompilation Documentation, External searches and Metalink z
Troubleshooting information gathering (example)
Java “Full thread dump” z z
How to get a Full thread dump? The Full thread dump interpretation Different
types of the threads within OC4J container Different states of the threads Some threads examples z z
Useful java runtime parameter Java DEADLOCK problem
How to get a “Full thread dump”? z Get
pid of the OC4J process
Use : ps, pstree or opmnadmin $ opmnadmin debug comp=pm HTTP/1.1 200 OK Content-Length: 677 Content-Type: text/html Response:
======== PM ======== PM requests processed: 5 PROCESS TABLE UID ---------456236 390700 521772 259628 194092 128556
PID ----13512 13511 15874 13415 13338 13281
FLAGS TYPE STATUS REF HTTP AJP RMI JMS -------- ------- ------- ------ ----- ----- ----- ----00000040 OC4J Alive 1 0 3003 3104 3204 00000040 OC4J Alive 1 0 3002 3103 3203 00000040 OC4J Alive 1 0 3001 3102 3202 00000040 OC4J Alive 1 0 3000 3101 3201 00000040 Apache Alive 1 7780 0 0 0 00000040 Generic Alive 1 0 0 0 0
How to get a “Full thread dump”? z
Find out the output file name (OH/opmn/logs/*) $ ps -fw --width 4000 -p 16114 UID PID PPID C STIME TTY TIME CMD oracle 15874 15902 0 23:40 ? 00:00:00 /u01/app/oracle/product/J2EE01/jdk/bin/i386/native_threads/java -Xms512M Xmx512M -Doracle.ons.oraclehome=/u01/app/oracle/product/J2EE01 Doracle.ons.oracleconfighome=/u01/app/oracle/product/J2EE01 Doracle.ons.clusterid=1 -Doracle.ons.instanceid=10.130.2.9.20dcd9.fdc4ac6949.8000 -Doracle.ons.indexid=PANDA.default_island.1 -DOPMN=true -jar oc4j.jar config /u01/app/oracle/product/J2EE01/j2ee/PANDA/config/server.xml -properties instance -vmId C_10.130.2.9.20dcd9.fdc4ac6949.8000#J2EE01.ap1.voava.lv#521772#PANDA#PANDA#default_island -ajp 3001 -rmi 3102 jms 3202 $ ps -wo args --width 4000 -p 15874 | grep -v COMMAND | awk -F"oracle.ons.indexid=" '{print $2}' | awk '{print $1}' PANDA.default_island.1
z
Send SIGQUIT (Quit from keyboard) signal $ kill -3 15874
How to get a “Full thread dump”? z
Check the output file
OH/opmn/logs/PANDA.default_island.1 Full thread dump: "Thread-13" daemon prio=1 tid=0x82c8690 nid=0x3f15 waiting on monitor [0xa51ff000..0xa51ff89c] at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:427) at java.util.TimerThread.run(Timer.java:380) "Thread-12" prio=1 tid=0x67b0cce0 nid=0x3e02 waiting on monitor [0..0xbffeb62c] "OC4JMonitorThread" daemon prio=1 tid=0x67691ec8 nid=0x3ef2 waiting on monitor [0xa53ff000..0xa53ff89c] at java.lang.Object.wait(Native Method) at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.java:251) at oracle.ons.NotificationQueue.dequeue(NotificationQueue.java:224) at oracle.ons.Subscriber.receive(Subscriber.java:136) at com.evermind.server.OC4JMonitorThread.run(OC4JMonitorThread.java:419) at java.lang.Thread.run(Thread.java:479) "Thread-10" daemon prio=1 tid=0x676930f0 nid=0x3ef1 waiting on monitor [0xa55ff000..0xa55ff89c] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:415) at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.java:253) at oracle.ons.NotificationQueue.dequeue(NotificationQueue.java:213) at oracle.ons.SenderThread.run(SenderThread.java:81)
The “Full thread dump” Interpretation ===================================================== "OC4JMonitorThread" daemon prio=1 tid=0x67691ec8 nid=0x3ef2 waiting on monitor [0xa53ff000..0xa53ff89c] at java.lang.Object.wait(Native Method) at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.java:251) at oracle.ons.NotificationQueue.dequeue(NotificationQueue.java:224) at oracle.ons.Subscriber.receive(Subscriber.java:136) at com.evermind.server.OC4JMonitorThread.run(OC4JMonitorThread.java:419) at java.lang.Thread.run(Thread.java:479)
===================================================== [OC4JMonitorThread] – name (type) of the thread OC4JMonitorThread.java Thread.currentThread().setName("OC4JMonitorThread");
[nid=0x3ef2] – Thread ID (in decimal = 16114) [waiting on monitor] – State of the thread at the snap time [oracle.ons.NotificationQueue.internalDequeue] – java stack
TYPES of the Threads OC4J Specific AJPConnectionListener [0.0.0.0/0.0.0.0:3001] 1 RMIServer [0.0.0.0/0.0.0.0:3102] count:1 1 RMIServer [0.0.0.0/0.0.0.0:3102] count:2 1 JMSServer 1 OC4JMonitorThread 1 TaskManager 1 Thread 5 ApplicationServerThread 195 JAVA GENERIC VM Thread 1 VM Periodic Task Thread 1 Reference Handler 1 Signal Dispatcher 1 Suspend Checker Thread 1 Finalizer 1 1
STATES of the Threads z
z
z
Waiting on monitor
In most of cases it is an idle state of the thread. The thread is waiting for a task to execute or voluntary sleeps
All the available ApplicationServerThreads are in this state
Runnable
The thread is executing application code or reading from the socket
All listeners must be in this state
Waiting for monitor entry
Waiting to lock an object (other thread is holding the lock). This happens if two or more threads are trying to execute synchronized java code None of threads must be in this state for a long time If the system thread stays in this state for a long time you are in trouble If many threads are in this state you have the situation of serialization
Threads Examples 1. OC4JMonitorThread is waiting for a work. Usual state for most of the threads "OC4JMonitorThread" daemon prio=1 tid=0x679a3518 nid=0x573 waiting on monitor [0xa53ff000..0xa53ff89c] at java.lang.Object.wait(Native Method) at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.java:251) at oracle.ons.NotificationQueue.dequeue(NotificationQueue.java:224) at oracle.ons.Subscriber.receive(Subscriber.java:136) at com.evermind.server.OC4JMonitorThread.run(OC4JMonitorThread.java:419) at java.lang.Thread.run(Thread.java:479)
2. "ApplicationServerThread-5" – idle worker thread, is waiting for a task "ApplicationServerThread-5" prio=1 tid=0x6794da88 nid=0x43f waiting on monitor [0xb15ff000..0xb15ff89c] at java.lang.Object.wait(Native Method) at EDU.oswego.cs.dl.util.concurrent.BoundedBuffer.poll(BoundedBuffer.java:170) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:768) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:802) at java.lang.Thread.run(Thread.java:479)
3. "Thread-9" is reading from the socket (connected to the ONS process, port 7100). This thread most of time is waiting for a data to be read from the socket "Thread-9" daemon prio=1 tid=0x67a02920 nid=0x571 runnable [0xa57ff000..0xa57ff89c] at java.net.SocketInputStream.socketRead(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:85) at oracle.ons.InputBuffer.readMoreData(InputBuffer.java:267) at oracle.ons.InputBuffer.getNextString(InputBuffer.java:222) at oracle.ons.ReceiverThread.run(ReceiverThread.java:225)
Threads Examples 4. "AJPConnectionListener� is waiting for new connections from the Apache processes "AJPConnectionListener [0.0.0.0/0.0.0.0:3001]" prio=1 tid=0x67911828 nid=0x439 runnable [0xb19ff000..0xb19ff89c] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:463) at java.net.ServerSocket.implAccept(ServerSocket.java:238) at java.net.ServerSocket.accept(ServerSocket.java:217) at com.evermind.server.http.AJPConnectionListener.run(AJPConnectionListener.java:58) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:804) at java.lang.Thread.run(Thread.java:479)
5. "ApplicationServerThread-5" is reading the request from the Apache process "ApplicationServerThread-77" prio=1 tid=0x8391048 nid=0x4a75 runnable [0xb4dff000..0xb4dff89c] at java.net.SocketInputStream.socketRead(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:85) at com.evermind.io.SingleReadBufferInputStream.readChunk(SingleReadBufferInputStream.java:116) at com.evermind.io.SingleReadBufferInputStream.read(SingleReadBufferInputStream.java:35) at com.evermind.server.http.AJPRequestHandler.initRequest(AJPRequestHandler.java:345) at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:131) at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:72) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:804) at java.lang.Thread.run(Thread.java:479)
Apache
Threads Examples 6. "ApplicationServerThread-71" is executing database query, waiting for the results to be retrieved "ApplicationServerThread-71" prio=1 tid=0x8379130 nid=0x4a6f runnable [0xb59fe000..0xb59ff89c] at java.net.SocketInputStream.socketRead(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:85) at oracle.net.ns.Packet.receive(Unknown Source) at oracle.net.ns.DataPacket.receive(Unknown Source) at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:931) at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:893) at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:375) at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1986) at oracle.jdbc.ttc7.TTC7Protocol.parseExecuteFetch(TTC7Protocol.java:1144) at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java:2484) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2850) at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:609) at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:537) at oracle.jbo.server.ViewObjectImpl.getQueryHitCount(ViewObjectImpl.java:1522) at oracle.jbo.server.QueryCollection.getEstimatedRowCount(QueryCollection.java:1530) at oracle.jbo.server.ViewRowSetImpl.getEstimatedRowCount(ViewRowSetImpl.java:1482) at oracle.jbo.server.ViewObjectImpl.getEstimatedRowCount(ViewObjectImpl.java:4596) at _common._comp._CMN__DataScrollerComponent._jspService(_CMN__DataScrollerComponent.java:115) at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:56) at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:317) at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:476) at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:390) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65) at oracle.security.jazn.oc4j.JAZNFilter.doFilter(JAZNFilter.java:283) at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:566) at com.evermind.server.http.ServletRequestDispatcher.include(ServletRequestDispatcher.java:119) at com.evermind.server.http.GetParametersRequestDispatcher.include(GetParametersRequestDispatcher.java:95) at com.evermind.server.http.EvermindPageContext.include(EvermindPageContext.java:280) at oracle.jbo.html.jsp.datatags.ComponentTag.doStartTag(ComponentTag.java:70) at _skd._d001._D2260F__Browse._jspService(_D2260F__Browse.java:249) at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:56) at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:317) at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:476) at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:390) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65) at oracle.security.jazn.oc4j.JAZNFilter.doFilter(JAZNFilter.java:283) at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:566) at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:306) at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:767) at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:166) at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:72) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:804) at java.lang.Thread.run(Thread.java:479)
DB
Useful Java Runtime Parameter z
-XX:+JavaMonitorsInStackTrace (opmn.xml) <java-option value="-server -Xms512M -Xmx512M -verbose:gc -XX:+JavaMonitorsInStackTrace"/>
z
The parameter wasnâ&#x20AC;&#x2122;t set
"ApplicationServerThread-18" prio=1 tid=0x692c40e0 nid=0x2f92 waiting for monitor entry [0xbc1ff000..0xbc1ff89c] at oracle.jbo.server.ViewRowSetIteratorImpl.setRangeSize(ViewRowSetIteratorImpl.java:376) at oracle.jbo.server.ViewRowSetImpl.setRangeSize(ViewRowSetImpl.java:1659) at oracle.jbo.server.ViewObjectImpl.setRangeSize(ViewObjectImpl.java:4626) at lv.vovaa.panda.common.datatags.DataSourceCmn.doStartTag(DataSourceCmn.java:189) at _nsk._nsk001._M2801F__Edit._jspService(_M2801F__Edit.java:150) at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:56) at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:317) at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:476) at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:390) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) ...
z
The parameter was set
"ApplicationServerThread-96" prio=1 tid=0x69d6c8c0 nid=0xdce waiting for monitor entry [0xb25ff000..0xb25ff89c] at oracle.jbo.server.ViewRowSetIteratorImpl.setRangeSize(ViewRowSetIteratorImpl.java:376) - waiting to lock <460e0338> (a oracle.jbo.JboSyncLock) at oracle.jbo.server.ViewRowSetImpl.setRangeSize(ViewRowSetImpl.java:1659) at oracle.jbo.server.ViewObjectImpl.setRangeSize(ViewObjectImpl.java:4626) at lv.vovaa.panda.common.datatags.DataSourceCmn.doStartTag(DataSourceCmn.java:189) - locked <45f722d0> (a lv.vovaa.panda.common.datatags.DataSourceCmn) at _nsk._nsk001._AP3220F__Tickets__browse._jspService(_AP3220F__Tickets__browse.java:105) at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:56) at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:317) at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:476) at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:390) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) ...
OC4J
Useful Java Runtime Parameter z
It is possible to determine the blocking thread easily
"ApplicationServerThread-55" prio=1 tid=0x69d512b8 nid=0xda4 runnable [0xb77fe000..0xb77ff89c] at java.net.SocketInputStream.socketRead(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:85) at oracle.net.ns.Packet.receive(Unknown Source) ... at oracle.jbo.server.QueryCollection.executeQuery(QueryCollection.java:549) at oracle.jbo.server.ViewObjectImpl.executeQueryForCollection(ViewObjectImpl.java:2600) - locked <460e0338> (a oracle.jbo.JboSyncLock) at lv.vovaa.panda.common.pbo.PboViewObjectImpl.pboExecuteQueryForCollection(PboViewObject.java:324) at lv.vovaa.panda.common.pbo.PboViewObjectImpl.executeQueryForCollection(PboViewObjectImp.java:309) at oracle.jbo.server.ViewRowSetImpl.execute(ViewRowSetImpl.java:521) ...
z
Then we can locate the source code for troubleshooting purposes
ViewObjectImpl.java
protected void executeQueryForCollection(Object obj, Object aobj[], int i) { synchronized(getSyncLock()) { int j = 0; if(InstrumentedEvent.isActive) j = InstrumentedEvent.startEvent(EventGroup.EXECUTE_QUERY, "ViewObject executeQueryForCollection " + getName()); ((QueryCollection)obj).executeQuery(aobj, i); if(InstrumentedEvent.isActive) InstrumentedEvent.endEvent(j); } }
Java DEADLOCK z
Information from the OC4J process “Full thread dump”
Full thread dump: "ApplicationServerThread-126" prio=1 tid=0x881f668 nid=0x5d59 waiting for monitor entry [0xa19ff000..0xa19ff89c] ... ... FOUND A JAVA LEVEL DEADLOCK: ---------------------------"ApplicationServerThread-58": waiting to lock monitor 0x80b9804 (object 0x4595b408, a java.lang.Object), which is locked by "ApplicationServerThread-51" "ApplicationServerThread-51": waiting to lock monitor 0x80b9564 (object 0x45955750, a com.evermind.server.http.EvermindHttpSession), which is locked by "ApplicationServerThread-58" JAVA STACK INFORMATION FOR THREADS LISTED ABOVE: -----------------------------------------------Java Stack for "ApplicationServerThread-58": ========== at oracle.jbo.http.HttpContainer.setValue(HttpContainer.java:352) - waiting to lock <4595b408> (a java.lang.Object)
z z z
JDK/JRE (before 1.5.X) is unable to solve DEADLOCK situations OPMN doesn’t detect such kind of situations You need to take action from outside
Gather troubleshooting information and restart OC4J process
Classification of Problem Areas OC4J JVM(java) PORT: 3001
4.
AJP JMSServer JMSServer Connection Listener
Apache
New Request as sig n
ApplicationServerThread ApplicationServerThread ApplicationServerThread ApplicationServerThread ApplicationServerThread TaskManager
1.
ApplicationServerThread
2.
3.
DB
ApplicationServerThread ApplicationServerThread ApplicationServerThread
JDK OS
1. OC4J container inside (infinitive cycles, waits on other threads, deadlocks) 2. Interface to the Apache processes 3. Interface to the database 4. 500 Internal Error
Java “Full thread dump” (conclusion) z
It is an extremely useful source of information for understanding OC4J internals and troubleshooting
z
Take a few snaps to see dynamics
z
You need to analyze whole OC4J process “Full thread dump” to make conclusions
z
Java on Java. The ideal person for such analysis is the person who has a good knowledge of java, used technologies, application framework as well as application running on the container
z
DBA/Administrator should ensure all necessary information to be available to application troubleshooting person
Log Files z
OC4J output file Located
at OH/opmn/log/<comp_name>.default_island.<proc_nu m> (example PANDA.default_island.1) Store standard output of OC4J process (JAVA DUMP, task manager info, debug info -Dajp.io.debug=true) It can be useful to add time stamp information * * * * * echo "Timestamp: `date`" >> /u01/app/oracle/product/J2EE01/opmn/logs/PANDA.default_island.1
z
OC4J default-web-access.log Located
at
OH/ j2ee/PANDA/log/PANDA_default_island_1/default-web-access.log
It
is an analogue of access_log for Apache
OS Monitoring Commands z
System wide: vmstat, iostat, uptime, pstree
z
Top resources consumers: top, ps -eo pcpu,pid,args | sort -k1n | tail -10
z
Particular process information
$ while [ 1 ] ; do ps -p 13285 -o pcpu | grep -v CPU ; sleep 1 ; done 25.3 25.3 25.3 $ while [ 1 ] ; do ps -p 19273 -o vsz,rss | grep -v VSZ ; sleep 1 ; done 1095372 258244 1095372 258244 1095372 258244 $ ps -wo args --width 4000 -p 6018
OS Tracing Commands z
strace - trace system calls and signals for particular process
$ strace â&#x20AC;&#x201C;s1000 -f -p 7495 read(19, $ strace â&#x20AC;&#x201C;s1000 â&#x20AC;&#x201C;f -p 27021 --- SIGSTOP (Stopped (signal)) --read(104, "\0224\0\2\0\0", 4096) send(104, "A", 1, 0) send(104, "B", 1, 0) send(104, "\0", 1, 0) send(104, "\3", 1, 0) send(104, "\6", 1, 0) send(104, "\37", 1, 0) send(104, "\372", 1, 0) read(104, "\0224\0\2\0\0", 4096) send(104, "A", 1, 0) send(104, "B", 1, 0) send(104, "\0", 1, 0) send(104, "\3", 1, 0) send(104, "\6", 1, 0) send(104, "\37", 1, 0) send(104, "\372", 1, 0) read(104, "\0224\0\2\0\0", 4096) ...
z
= = = = = = = = = = = = = = = = =
6 1 1 1 1 1 1 1 6 1 1 1 1 1 1 1 6
pstack - print a stack trace of running processes
Apache Side Information z
Apache Server Status page
http://ap1.voava.lv:7780/server-status?refresh=15
z
log files  Â
Apache log files $OH/Apache/Apache/logs Sometimes information can mislead
access_log 192.168.2.12 - - [01/Feb/2005:16:43:34 +0200] "GET /PANDA/NSK/NSK003/NS0291F_Browse.jsp HTTP/1.1" 500 0 erorr_log [Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0080: After calling Apache's ap_bwrite(), got an return value: -1 and the client could have aborted in the middle of receiving the chunk data. [Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0058: Failed to handle response chunk data passed in from oc4j via ajp13. [Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0035: After hdndling ajp13 response message, got an invalid value: -2. [Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0121: Failed to service request with network worker: PANDA_15 and it is not recoverable. [Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0013: Failed to call destination: PANDA's service() to service the request.
Java Code Decompilation z
Extremely useful for troubleshooting purposes
z
OC4J
z
OH/dcm/lib/dcm.jar OH/dcm/lib/oc4j_deploy_tools.jar OH/dcm/lib/oc4j_remote_deploy.jar
DMS
z
OH/jdbc/lib/classes12.zip OH/jdbc/lib/classes12dms.jar
DCM
z
OH/j2ee/home/oc4j.jar
JDBC
z
Unzip archive (jar/zip) Decompile *.class files (http://www.kpdus.com/jad.html)
OH/lib/dms.jar
BC4J
OH/BC4J/redist/bc4j.ear OH/BC4J/jlib/* OH/BC4J/lib/*
Java Code Decompilation Usage Example z
Decompile all oc4j.jar classes z Grep through *.java files for “System.getProperty” 9.0.4 oc4j.jar … opmnPingInterval opmn.query.debug oracle.arraylist.deepCopy oracle.aurora.jem.scheduling.threads oracle.dms.console.DMSConsole oracle.dms.gate oracle.dms.sensors oracle.dms.transtrace.ecidenabled oracle.home oracle.ias.jcache
Debugging Switches z
Debug switches are documented in
Containers for J2EE User's Guide 10g Release 2 (10.1.2) Part No. B14011-01 B Additional Information B.8 OC4J Command-Line Options and System Properties z
Debug information example (-Dajp.io.debug=true)
PANDA.default_island.1 1107387907360:ThrP[50] ThrQ[0] 1107387907360:CxP[20] CxQ[0] - WebC[0] RmiC[0] AJP: Input request length is: 1 AJP: AJPOutputStream::write() of 1 AJP: AJPOutputStream::writeChunk(): 1 1107387908370:ThrP[50] ThrQ[0] 1107387908370:CxP[20] CxQ[0] - WebC[0] RmiC[0]
z
Some of available categories are : ajp, http, jms, rmi, jdbc
Documentation and Internet z z z
The information in 9.0.X documentation is very limited Oracle AS Documentation Library 10g Release 2 (10.1.2) Highly Recommended books
Containers for J2EE User's Guide (144) Containers for J2EE Standalone User's Guide (134) Server Administrator's Guide (542) OPMN Administrator's Guide (158) DCM Administrator's Guide (116) Server Performance Guide (234) z z z z z
10g Best Practices 10g (9.0.4) Part No. B12223-01 BC4J JDeveloper Online Help system. www.google.com http://otn.oracle.com http://metalink.oracle.com
Metalink Patches z
Look through available patches.
………………………………………………………………..
Metalink Patches z
Strongly recommended in case of 9.0.3 OC4J
Metalink Patches z
Monitor for new patches, which can be relative to your environment
Troubleshooting information gathering (example)
Performance tuning
Our Findings (Java Workers) z
Preset java workers threads for OC4J process
OH/j2ee/<comp_name>/config/server.xml <global-thread-pool min="100" max="300" queue="30" keepAlive="-1" cx-min="100" cx-max="300" cx-queue="30" cx-keepAlive="-1" debug="true"/>
Set min to high value. It allows OC4J not to create the additional threads on the fly
You don’t have to take care on max, keepAlive and queue parameters then OC4J doesn’t need to manage queue
debug="true" – Allows to see statistics in the OC4J process standard output file
It is useful as troubleshooting information as well. Record appears in the output file at the end of each Task manager cycle.
<PANDA.default_island.1> 1107390315893:ThrP[100] ThrQ[0] 1107390315893:CxP[100] CxQ[0] - WebC[0] RmiC[0] 1107390316903:ThrP[100] ThrQ[0] 1107390316903:CxP[100] CxQ[0] - WebC[0] RmiC[0] ...
Our Findings (OC4J Processes) z
Use many OC4J processes without enabled state replication
OH/opmn/conf/opmn.xml <oc4j maxRetry="3" instanceName="PANDA" gid="PANDA" numProcs="1"> <config-file path="/u01/app/oracle/product/J2EE01/j2ee/PANDA/config/server.xml"/> <java-option value="-server -Xms512M -Xmx512M"/> <oc4j-option value="-properties "/> <port ajp="3001-3100" jms="3201-3300" rmi="3101-3200"/> <island id="default_island" numProcs="3"/> </oc4j>
z z z z z
Initially it was recommended by Oracle support while TAR resolution OC4J container is more stable on a lower users load mod_oc4j distributes requests through 3 OC4J identical containers 1/3 of users load is handled by each OC4J process Do not use state replication
z
It can add significant load on CPU It complicates configuration It can lead to opposite effect
Be careful
You have to test this configuration for your application. It can lead to unpredictable DB connection behavior Memory/threads configuration is for each process of the OC4J instance
Our Findings (Memory) OH/opmn/conf/opmn.xml <oc4j … <java-option value="-server -Xms512M -Xmx512M"/> </oc4j> z Set max=min
z
JVM doesn’t need to manage memory allocation dynamically You have reserved memory on the server
Java Garbage collector
There is a lot of information around
z
http://developers.sun.com/techtopics/mobility/midp/articles/garbagecollection2/ http://www.cons.org/cmucl/doc/gc-tuning.html
-verbose:gc
Java Profilers
Add significant overhead to be used in production environments Extremely useful in test/dev environments for troubleshooting
Our Findings (Memory) z
Monitor memory usage in production
grep log files for “out of memory” error use OS utilities use dmstool
$ dmstool -table JVM | grep 3001 | grep Memory /ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.value /ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.minValue /ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.maxValue /ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.value /ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.minValue /ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.maxValue ap1.voava.lv*J2EE01-/u01/app/oracle/product $
495796 kbytes 495796.0 kbytes 522468.0 kbytes 524032 kbytes 524032.0 kbytes 524032.0 kbytes
Our Findings (Stresstesting) z z
The best way to tune performance is a stresstesting According to our experience
After Development had performed the performance testing, a lot of problems were solved An automatic testing tool has been used (Rational Robot) Different modules were tested Different load profiles were tested Testing was repeated several times until acceptable results were achieved
Response Time Monitoring by DMS
Apache OHS [c]
Send
Request B
OC4J mod_plsql handle Handle B
JVM [java]
ApplicationServerThread Read Resolve
Process B JSP Processing Servlet Processing
mod_oc4j handle
Process E Handle E Handle B Handle E
Request E
Recive
mod_perl mod_php ...
DMS Sensors
DMS Sensors
DMS-client DMS.JAR (java) statistics reflecting
DB
Response Time Breakdown Example $ dmstool -table ohs_server | grep time .../Apache/connection.time 48123608767 usecs .../Apache/request.time 15916131942 usecs .../Apache/handle.time 15913988395 usecs rhoracle.voava.lv*J2EE01-/u01/app/oracle/product/J2EE01/Apache/Apache/conf $ $ dmstool -table ohs_module | grep time | sort -k2n | tail -2 .../Apache/Modules/mod_dms.c/handle.time 8081655 usecs .../Apache/Modules/mod_oc4j.c/handle.time 15905072698 usecs $ $ dmstool -table oc4j_jspExec | grep time .../oc4j/PANDA/WEBs/PANDA/JSPs/processRequest.time 17 563 793 msecs $ $ dmstool -table "oc4j_jsp(threadsafe=true)" | grep time | sort -k2n | tail -5 .../oc4j/PANDA/WEBs/PANDA/JSPs/NSK/NSK003/NS0285F_searchForm.jsp/service.time 1 072 570 msecs .../oc4j/PANDA/WEBs/PANDA/JSPs/SKD/d001/D2261F_PersonsBrowse.jsp/service.time 1 407 663 msecs .../oc4j/PANDA/WEBs/PANDA/JSPs/NSK/NSK003/NS0291F_Browse.jsp/service.time 2 942 655 msecs .../oc4j/PANDA/WEBs/PANDA/JSPs/SKD/d001/D2261F_Edit.jsp/service.time 3 151 184 msecs .../oc4j/PANDA/WEBs/PANDA/JSPs/SKD/d001/D2260F_Browse.jsp/service.time 4 847 408 msecs rhoracle.voava.lv*J2EE01-/u01/app/oracle/product/J2EE01/Apache/Apache/conf $
DMS Sensors Classification z
Informative sensors – reflect a configuration information and some system wide statistics
z
Average performance sensors – reflect response time totals by modules, applications, etc.
z
ohs_child, ohs_responses, JVM, oc4j_ear, oc4j_opmn, oc4j_task, JDBC_Driver , JDBC_Connection
ohs_server , ohs_module, ohs_virtualHost, oc4j_context, oc4j_web_module, oc4j_jspExec
Units performance sensors – reflect response time statistics for each unit
oc4j_servlet oc4j_jsp(threadsafe=true)
Collecting and Reflecting DMS Statistics z z z
DMS Sensors statistics is incremental Make periodic snaps of statistics for further analysis Develop interface to access statistics easily Â
Graphical interface for system wide statistics
Â
Textual interface for units (JSP/Servlet) response time analysis
Collecting and Reflecting DMS Statistics SYS:PROD> @sp_jsp Completed Snapshots Snap Id ------530 531 532 549
Snap Started ------------------2005-02-03 12:33:26 2005-02-03 13:03:24 2005-02-03 13:33:27 2005-02-03 14:03:32
Specify the Begin and End Snapshot Ids ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Enter value for begin_snap: 532 Enter value for end_snap: 549 Saved in c:\temp\532_549.log Snap Id Snap Time ------- -----------------Begin Snap: 532 2005-02-03 13:33:27 End Snap: 549 2005-02-03 14:03:32 Elapsed: 30.08 (mins) JSP ordered by Response time JSP Name Response Execution Average (sec) Time(sec) Count Response ------------------------------------------------------------------ --------- ---------- ---------../3001/../NSK/NSK001/AP3220F_TicketImportProcessOneByOne.jsp 956.439 2 478.22 ../3002/../NSK/NSK001/AP3220F_Tickets_browse.jsp 708.967 71 9.99 ../3001/../NSK/NSK001/M2801F_Browse.jsp 533.389 74 7.21 ../3003/../NSK/NSK001/P2600F_XMLImportProcess.jsp 471.106 1 471.11 ../3003/../LRG/l001/L3000F_Browse.jsp 293.775 92 3.19 ../3002/../NSK/NSK001/AP3221F_ApnAccountingDocFilter.jsp 291.382 6 48.56 ../3002/../common/comp/CMN_DataFilterComponent.jsp 238.75 35 6.82 ../3001/../NSK/NSK001/SP0301F.jsp 216.499 32 6.77 ../3001/../SKD/d001/D2262F_Browse.jsp 208.709 7 29.82 ../3001/../NSK/NSK001/SP0301F_filter.jsp 168.957 6 28.16 ../3002/../SKD/d001/D2260F_Browse.jsp 138.453 45 3.08 ../3003/../SKD/d001/D2260F_Browse.jsp 81.773 24 3.41 ../3001/../NSK/NSK001/Z3601F_Save.jsp 80.127 30 2.67 ../3001/../common/comp/CMN_LOVcomp.jsp 73.544 52 1.41 ../3003/../NSK/NSK001/SP0301F.jsp 72.548 29 2.5 ../3001/../SKD/d001/D2260F_Browse.jsp 71.642 62 1.16 ../3002/../NSK/NSK001/AP3221F_Save.jsp 68.509 91 .75 ../3001/../common/comp/CMN_DataHandlerComponent.jsp 68.46 765 .09 ../3002/../common/comp/CMN_DataHandlerComponent.jsp 57.144 722 .08 SYS:PROD>
Monitor All the Tiers of the System z
Do not forget to monitor OS
and HW resources utilization Network Database
Instrument Your Code z
AskTom.oracle.com
http://asktom.oracle.com/pls/ask/f?p=4950:8:16675873436862412624::NO::F4950_P8_DISPLAYID,F4950_P8_CRITERIA:12836314571537
â&#x20AC;Ś Another example is asktom.oracle.com. Just go there and click on any article from the home page. You'll see a URL similar to: http://asktom.oracle.com/pls/ask/f?p=...::NO:: If you simply type over that URL and replace the word NO with YES - you'll see the same page but with lots and lots of state/timing information dumped into it. â&#x20AC;Ś
BC4J
OC4J Internals & BC4J ONS
PORT: 3001
Thread
AJP Connection Listener
OC4JMonitor Thread
new req ApplicationServerThread
Thread
ApplicationServerThread
ApplicationServerThread ApplicationServerThread
Java native
ApplicationServerThread ...
AJP 1.3
ONet
ApplicationServerThread ApplicationServerThread
Memory Structures JBO Application Module Pool JBO Application Module Pool JBO Application Module Pool JBO Application Module Pool
JBO Connection pool Application objects JBO View Objects
3101
RMI RMIServer Server
3201
Http Session List
JMSServer
PORT: 3101
PORT: 3201
...
TaskManager
TaskManager JMSServer JMSServer JMSServer JMSServer Thread Thread
...
Java native VM Thread VM Thread Signal Dispatcher Signal Dispatcher Suspend Checker Suspend ThreadChecker Thread
VM Periodic Task VMThread Periodic Task Thread Finalizer Finalizer Reference Reference Handler Handler
Usage Considerations z z z z
Most information is available under JDeveloper cover Use development version of BC4J Deploy BC4J.EAR to your OC4J component (do not use home component) There is the BC4J monitoring page
Useful Information Sources 0. 1. 2. 3. 4.
Metalink Note:273350.1 BC4J configuration tuning tips Metalink Note:283476.1 BC4J (ADF BC) Configuration Tuning Tips Metalink Note:238634.1 JBO / BC4J Parameters And Defaults Metalink Note:231141.1 BC4J Row Spillover and tuning How to Performance Tune an ADF Business Components (ADF BC) Application September 23, 2004 Steve Anderson and Michael Gantman http://www.oracle.com/technology/products/jdev/tips/muench/ampooling/index.html 5. Understanding Application Module Pooling Concepts and Configuration Parameters Author: Steve Muench, BC4J Development Team Date: August 5, 2004 http://www.oracle.com/technology/products/jdev/tips/muench/ampooling/index.html 6. Getting Application Module Pool Statistics to Aid with Size Tuning By Steve Muench http://radio.weblogs.com/0118231/stories/2004/05/10/gettingApplicationModulePoolStatisticsToAidWithSizeTuning.html 7. Overview of Temporary Tables Created By BC4J An Oracle technical white paper April 2002 http://www.oracle.com/technology/products/jdev/htdocs/bc4j/bc4j_temp_tables.html 8. How To Support Dynamic JDBC Credentials An Oracle BC4J technote Version 3: April 23rd 2002 http://www.oracle.com/technology/products/jdev/howtos/bc4j/howto_dynamic_jdbc.html 9. BC4J/JClient Performance Study Written by Steve Muench, Oracle Corporation January, 2004 http://www.oracle.com/technology/products/jdev/tips/muench/jclientperf/index.html 10. Dive into BC4J and ADF Tips and tricks from Steve Muench on using Oracle's J2EE frameworks, JDeveloper Java IDE, and XML technologies... http://radio.weblogs.com/0118231/
Performance Tuning Parameters jbo.pers.max.active.nodes=1000 jbo.pers.max.rows.per.node=1000 jbo.doconnectionpooling=true jbo.poolminavailablesize=0 jbo.poolmaxavailablesize=600 jbo.poolmonitorsleepinterval=600000 jbo.poolmaxinactiveage=600000 jbo.ampool.monitorsleepinterval=300000 jbo.ampool.maxinactiveage=600000
JBO Parameter lookup order: Set programmatically or configuration in bc4j.xcfg file) Applet tags -D flags (or oc4j.properties file) BC4J.properties file in current working directory of Java VM /oracle/jbo/BC4J.properties resource in the runtime class path /oracle/jbo/server/jboserver.properties resource (by default found inside bc4jmt.jar) /oracle/jbo/common/Diagnostic.properties resource (by default found inside bc4jmt.jar) System defined default as a fallback
Summary
Summary z z z z z z
Introduction Usage considerations OC4J Internals OC4J Troubleshooting OC4J Performance tuning BC4J
Q&A