Oracle ERP DBA

Today morning, we (the DBAs) were faced with a strange issue. On Saturday (the 14th of February - (St.) Valentine's day!!), the UNIX boxes hosting our critical test instance went down for scheduled maintenance without any prior information. So, we could not bring down our databases and applications. As soon as the unix box came up, we encountered this issue. Let me briefly describe the environment, the issue, the analysis performed and of course, the solution to this most vexing problem. Environment : Apps Version : 11.5.10.2 with ATG RUP 5 DB Version : 10.2.0.3 Architecture : Two Node instance with database + admin on one tier and web server on another. PS : Another instance (development) exists on the same server with the same configuration, which was working fine. Issue : When trying to access the login page of Oracle Apps, received the below error. Even OAM login page was throwing the same error. Request URI: /OA_HTML/AppsLocalLogin.jsp Exception: java.lang.ExceptionInIniti

When passing a variable in unix, the general convention is to use $variable . Although this does suffice in most cases but in some rare cases, it gives connotations totally different than what was originally intended, as shown in the below example. The requirement Copy all files ending with ver to ver_old. For example, suppose I have a file called formver, I would like to copy it to formver_old. The code below is the code that I wrote to achieve the above requirement. for i in `ls *ver` do cp -p $i $i_old done The pitfall A cursory glance at the code indicates that it should run smoothly like a river. But it doent' and herein lies the catch. Let's have a look at the error and figure out what exactly is wrong with the above code. __________________________________________________________________ $ for i in `ls *ver` > do > cp -p $i $i_old > done cp: Insufficient arguments (1) Usage: cp [-f] [-i] [-p] [-@] f1 f2 cp [-f] [-i] [-p] [-@] f1 ... fn d1 cp -r-R [-H-L-P] [-f] [

Apps Configuration Oracle Apps Version ==> 11.5.9 Node Type ==> Multi-node RAC ==> No OS & version ==> Sun SPARC Solaris (64-Bit) 10 (5.10) Issue Recently (last week, to be precise), I was researching an "Internal Server error" while opening Installed Base. The error log had error "client denied by server configuration" (not very helpful, I would say). The mod_jserv.log was much more informative and in fact, quite specific. Refer the note marked in red below. __________________________________________________________________ [28/01/2009 06:39:44:223] (ERROR) ajp12: Servlet Error: OracleJSP: oracle.jsp.provider.JspCompileException: <H3>Errors compiling$COMMON_TOP/_pages/_oa__html//_csiSwitchRespMain.java< /H3><TABLE BORDER=1 WIDTH=100%><TR><TH ALIGN=CENTER>Line #</TH><TH ALIGN=CENTER>Error</TH></TR><TR><TD WIDTH=7% VALIGN=TOP><P ALIGN=CENTER>990</TD><TD> as of release

Listed below is the ready reckoner for finding out the kernel bit information for Linux, Solaris, HP-UX and IBM AIX, which are the most commonly used OS for Oracle Apps. Linux getconf LONG_BIT or uname -m (works for x86-64 bit servers. Not tested it in 32 bit servers) eg: $ getconf LONG_BIT 64 $ uname -m x86_64 $ Sun SPARC Solaris isainfo -kv eg: $ isainfo -kv 64-bit sparcv9 kernel modules $ HP-UX getconf KERNEL_BITS eg: $ getconf KERNEL_BITS 64 $ IBM AIX bootinfo -K Not tested as I currently do not have any AIX servers to test on! Note 1 : To find whether a given executable is 32-bit or 64-bit, use the command file <executable_name> eg: $ file oracle oracle: ELF 64-bit MSB executable SPARCV9 Version 1, dynamically linked, not stripped $ file tnslsnr tnslsnr: ELF 64-bit MSB executable SPARCV9 Version 1, dynamically linked, not stripped $ file sqlplus sqlplus: ELF 64-bit MSB executable SPARCV9 Version 1, dynamically linked, not stripped Note 2 : You can install a 32-bit executab

Case Facts : Database Version ==> 10.2.0.3.0 Oracle Apps Version ==> 11.5.10.2 Node Type ==> Multi-node RAC ==> No OS & version ==> Sun SPARC Solaris (64-Bit) 10 (5.10) Symptom ==> Database crashes with ORA-07445 Case Description : While trying to bring up an Oracle Apps instance, I found that the database was crashing with ORA-07445 errors (as below) as soon as I brought up the concurrent managers. _____________________________________________________________ Mon Feb 2 00:26:24 2009 Errors in file $ORACLE_HOME/admin/[CONTEXT_NAME]/udump/cpek_ora_6085.trc: ORA-07445: exception encountered: core dump [lfilic()+328] [SIGSEGV] [Address not mapped to object] [0x000000040] [] [] ORA-29282: invalid file ID Mon Feb 2 00:26:27 2009 Errors in file $ORACLE_HOME/admin/[CONTEXT_NAME]/udump/cpek_ora_5512.trc: ORA-07445: exception encountered: core dump [lfilic()+328] [SIGSEGV] [Address not mapped to object] [0x000000040] [] [] Mon Feb 2 00:26:34 2009 Errors in file $ORACLE_HOM

I was trying to find something in the CONTEXT_FILE when I happened to notice the AF_CLASSPATH env variable (not that I haven't noticed it earlier). But this time, it set me thinking as to what might its purpose be since we already had a CLASSPATH variable and its value seemed to be the same as the AF_CLASSPATH variable. I found 3 variables starting with AF: AF_JRE_TOP AF_CLASSPATH AFJVAPRG. I wanted to find out what they were and why they were used. This is the explanation provided in Metalink Note 412709.1 (Oracle Workflow Documentation Updates for 11i.ATG_PF.H.delta.5 (RUP 5)) AFJVAPRG - The location of the JDK or JRE executable for the concurrent processing tier. AF_CLASSPATH - The classpath for the concurrent processing tier. Also, Note 373386.1 has this to say about AF_CLASSPATH. The AF_CLASSPATH variable is used by JAVA concurrent programs and must use literal (full) path values. By combining the above two notes, we can perhaps safely say that the AF variables are set for l