Monday, February 16, 2009

java.lang.ExceptionInInitializerError while trying to Access login page

Today morning, we (the DBAs) were faced with a strange issue. On Saturday (the 14th of February - (St.) Valentine's day!!), the UNIX boxes hosting our critical test instance went down for scheduled maintenance without any prior information. So, we could not bring down our databases and applications.


As soon as the unix box came up, we encountered this issue. Let me briefly describe the environment, the issue, the analysis performed and of course, the solution to this most vexing problem.


Environment:

Apps Version: 11.5.10.2 with ATG RUP 5

DB Version: 10.2.0.3

Architecture: Two Node instance with database + admin on one tier and web server on another.

PS: Another instance (development) exists on the same server with the same configuration, which was working fine.



Issue:

When trying to access the login page of Oracle Apps, received the below error. Even OAM login page was throwing the same error.

Request URI: /OA_HTML/AppsLocalLogin.jsp
Exception:

java.lang.ExceptionInInitializerError

Request URI:/OA_HTML/AppsLocalLogin.jsp
Exception:

java.lang.NoClassDefFoundError

As soon as the services were bounced, we were receiving the first error. After some time, we received the second error (yes, truth is indeed stranger than fiction!!).



Analysis:

Since, we were receiving "NoClassDefFoundError", performed the following:

1. Checked the CLASSPATH and AF_CLASSPATH variable and ensured that every path is valid.



2. Checked all the class paths defined by wrapper.classpath in jserv.properties using the command below:

for i in `grep classpath $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties grep -v "#" cut -d '=' -f2`
do
ls -ld $i >> /tmp/check_class_path.txt
if [[ $? -eq 0 ]]; then
continue
else
printf "\n\n '${i}' is not proper. \n\n"
fi
done


I compared the output generated from the above script with that from the development instance that was working and found no discrepancies.



3. Ran aoljtest.jsp and found that it was running fine till the step "Verify Guest User", where it was throwing "Page cannot be found" error.



4. The above error set me thinking and I checked the GUEST username and password in the

  • dbc file
  • database (profile option)
  • CONTEXT_FILE
  • output of the below query "select fnd_web_sec.validate_login ('GUEST','<PASSWORD>') from dual;"

and found it to be consistent.



5. Checked a lot of Metalink Notes and documents and blogs and what not, but to no avail.



6. Enabled debug in :

  • jserv.conf
  • jserv.properties
  • httpd.conf

but no log file was informative enough to point me in the right direction.



7. Meanwhile, the error log had been throwing the below error, which I analysed but could not make much headway and found it to be actually misleading.

[Sun Feb 15 23:13:20 2009] [error] [client <IP_ADDRESS>] (78)File name too long: access to /redirectURL$url=http_3A$$ <hostname.domain_name>_3A<port_number>$oa*_servlets$ oracle.apps.fnd.sso.AppsLogin_3FrequestUrl$=$oa*_servlets$weboam$ oam$oamLogin_26cancelUrl$=http_3A$$ <hostname.domain_name>_3A<port_number>$oa*_servlets$ oracle.apps.fnd.sso.AppsLogin$forward=oam$oamLogin failed


8. After exhausting all the options, I decided to take a break and promptly headed for lunch!



9. Post lunch, I tried to re-examine the issue from a different perspective. As point 3 above states, I was receiving an error while trying to check the Guest username using aoljtest.jsp. Again checked the guest username and password in the palces mentioned in point 4 above and found them to be consistent.



10. Finally (actually, I had found this note earlier, but ignored it because I found the guest username and password to be correct), I found Note: 458064.1, which I found relevant on three counts:

  • It mentions about the "java.lang.ExceptionInInitializerError" error.
  • we were also on ATG RUP 5 (5473858), as the note mentions.
  • It also mentions that "Verify Guest user" ends with an error (though different from the one that I was receiving).

So, I decided to follow the solution mentioned in that note.



Solution:

1. a. Login directly to Forms via http://<hostname>:<port>/dev60cgi/f60cgi

b. Set the value of system profile "Password Case Option" to "Insensitive"

c. Save and close Oracle Applications


2. On the operating system as the apps user (e.g. applmgr) execute these commands:

a. cd $FND_TOP/secure/<SID>_<hostname>

b. java oracle.apps.fnd.security.AdminAppServer <apps username>/<apps password> UPDATE GUEST_USER_PWD=GUEST/<PASSWORD ALL UPPERCASE> DB_HOST=<host> DB_NAME=<SID> DB_PORT=<port>


The values for GUEST_USER_PWD, DB_HOST, DB_NAME, and DB_PORT can be found in the $FND_TOP/secure/<SID>_hostname/<SID>.dbc file if required.


3. Verify that the following things are true:

a. The GUEST password is updated and all upper-case in the $FND_TOP/secure/<SID>_<hostname>/<SID>.dbc file.

b. That the profile GUEST_USER_PWD is set correctly after the password change. The SQL to run is:

set serveroutput on
set echo on
set timing on
set feedback on
set long 10000
set linesize 120
set pagesize 132
column SHORT_NAME format A30
column NAME format A40
column LEVEL_SET format a15
column CONTEXT format a30
column VALUE format A60 wrap
select p.profile_option_name SHORT_NAME,
n.user_profile_option_name NAME,
decode(v.level_id,
10001, 'Site',
10002, 'Application',
10003, 'Responsibility',
10004, 'User',
10005, 'Server',
10007, 'SERVRESP',
'UnDef') LEVEL_SET,
decode(to_char(v.level_id),
'10001', '',
'10002', app.application_short_name,
'10003', rsp.responsibility_key,
'10005', svr.node_name,
'10006', org.name,
'10004', usr.user_name,
'10007', 'Serv/resp',
'UnDef') "CONTEXT",
v.profile_option_value VALUE
from fnd_profile_options p,
fnd_profile_option_values v,
fnd_profile_options_tl n,
fnd_user usr,
fnd_application app,
fnd_responsibility rsp,
fnd_nodes svr,
hr_operating_units org
where p.profile_option_id = v.profile_option_id (+)
and p.profile_option_name = n.profile_option_name
and upper(n.user_profile_option_name) = 'GUEST_USER_PWD'
and usr.user_id (+) = v.level_value
and rsp.application_id (+) = v.level_value_application_id
and rsp.responsibility_id (+) = v.level_value
and app.application_id (+) = v.level_value
and svr.node_id (+) = v.level_value
and org.organization_id (+) = v.level_value
order by short_name, level_set;


c. That the following SQL returns 'Y':

SELECT fnd_web_sec.validate_login('GUEST','') FROM dual;


4. Bounced the services and retried the login, which worked like a charm.

Thursday, February 5, 2009

Flower brackets {} and variables in UNIX - A perfect match

When passing a variable in unix, the general convention is to use $variable. Although this does suffice in most cases but in some rare cases, it gives connotations totally different than what was originally intended, as shown in the below example.


The requirement
Copy all files ending with ver to ver_old. For example, suppose I have a file called formver, I would like to copy it to formver_old.


The code
below is the code that I wrote to achieve the above requirement.

for i in `ls *ver`
do
cp -p $i $i_old
done


The pitfall
A cursory glance at the code indicates that it should run smoothly like a river. But it doent' and herein lies the catch. Let's have a look at the error and figure out what exactly is wrong with the above code.
__________________________________________________________________

$ for i in `ls *ver`
> do
> cp -p $i $i_old
> done
cp: Insufficient arguments (1)
Usage: cp [-f] [-i] [-p] [-@] f1 f2
cp [-f] [-i] [-p] [-@] f1 ... fn d1
cp -r-R [-H-L-P] [-f] [-i] [-p] [-@] d1 ... dn-1 dn
cp: Insufficient arguments (1)
Usage: cp [-f] [-i] [-p] [-@] f1 f2
cp [-f] [-i] [-p] [-@] f1 ... fn d1
cp -r-R [-H-L-P] [-f] [-i] [-p] [-@] d1 ... dn-1 dn
cp: Insufficient arguments (1)
Usage: cp [-f] [-i] [-p] [-@] f1
cp [-f] [-i] [-p] [-@] f1 ... fn
cp -r-R [-H-L-P] [-f] [-i] [-p] [-@] d1 ... dn-1 dn
cp: Insufficient arguments (1)
Usage: cp [-f] [-i] [-p] [-@] f1
cp [-f] [-i] [-p] [-@] f1 ... fn
cp -r-R [-H-L-P] [-f] [-i] [-p] [-@] d1 ... dn-1 dn
cp: Insufficient arguments (1)
Usage: cp [-f] [-i] [-p] [-@] f1
cp [-f] [-i] [-p] [-@] f1 ... fn
cp -r-R [-H-L-P] [-f] [-i] [-p] [-@] d1 ... dn-1 dn
cp: Insufficient arguments (1)
Usage: cp [-f] [-i] [-p] [-@] f1
cp [-f] [-i] [-p] [-@] f1 ... fn
cp -r-R [-H-L-P] [-f] [-i] [-p] [-@] d1 ... dn-1 dn
__________________________________________________________________

Can you guess or pinpoint as to what exactly is wrong with the above code, which makes it throw this error?

It took me sometime but I finally figured it out. $i_old was being considered as a variable i.e., the shell was reading the variable as $i_old and not as the variable $i appended with an underscore(_) and 'old'.


The solution
The solution to this obviously is to use the flower brackets (braces/parantheses) for the variable $i so as to make it ubiquitous. Now the code looks like:
for i in `ls *ver`
do
cp -p ${i} ${i}_old
done


The conclusion
To conclude, I would like to say to my fellow DBAs who are also into shell scripting, watch out for small things like these, which might make or break your day!

Wednesday, February 4, 2009

11.5.9 & JDK 5 - Mutually exclusive

Apps Configuration

Oracle Apps Version ==> 11.5.9
Node Type ==> Multi-node
RAC ==> No
OS & version ==> Sun SPARC Solaris (64-Bit) 10 (5.10)

Issue

Recently (last week, to be precise), I was researching an "Internal Server error" while opening Installed Base.

The error log had error "client denied by server configuration" (not very helpful, I would say). The mod_jserv.log was much more informative and in fact, quite specific. Refer the note marked in red below.
__________________________________________________________________

[28/01/2009 06:39:44:223] (ERROR) ajp12: Servlet Error: OracleJSP: oracle.jsp.provider.JspCompileException: <H3>Errors compiling$COMMON_TOP/_pages/_oa__html//_csiSwitchRespMain.java<
/H3><TABLE BORDER=1 WIDTH=100%><TR><TH ALIGN=CENTER>Line #</TH><TH ALIGN=CENTER>Error</TH></TR><TR><TD WIDTH=7% VALIGN=TOP><P ALIGN=CENTER>990</TD><TD> as of release 1.5, 'enum' is a keyword, and may not be used as an identifier (try -source 1.4 or lower to use 'enum' as an identifier) Enumeration enum = request.getParameterNames(); </TD></TR><TR><TD WIDTH=7% VALIGN=TOP><P ALIGN=CENTER>991</TD><TD> as of release 1.5, 'enum' is a keyword, and may not be used as an identifier (try -source 1.4 or lower to use 'enum' as an identifier) while (enum.hasMoreElements()) </TD></TR><TR><TD WIDTH=7% VALIGN=TOP><P ALIGN=CENTER>993</TD><TD> as of release 1.5, 'enum' is a keyword, and may not be used as an identifier (try -source 1.4 or lower to use 'enum' as an identifier) String param = (String)enum.nextElement();
__________________________________________________________________

Research

Now, the above error gave me an idea. I decided to check the Metalink Note 300482.1 - Overview of Using Java with Oracle E-Business Suite Release 11i.

This note clearly states that for 11.5.9, the JDK (Java SE) version supported is 1.4.2.

Does that mean that JDK 5 and JDK 6 are not supported for 11.5.9? Not quite, as I found out.

The Note 304099.1 (Using J2SE Version 5.0 with Oracle E-Business Suite 11i) does NOT mention 11.5.9 anywhere, implying that JDK 5 is indeed not supported with 11.5.9.

However, the Note 401561.1 (Using J2SE Version 6 with Oracle E-Business Suite 11i) does mention that JDK 6 can be used with 11.5.9 provided (yes, as the ads say - conditions apply!) that you "ensure that your E-Business Suite Release 11i system has been upgraded to Maintenance Pack 11.5.9 CU2 or higher with ATG_PF.H RUP5 or higher on the 11.5.9 stream."

Conclusion

This means that 11.5.9 is indeed supported with JDK 6 but NOT JDK 5 (of course, provided that the above conditions are satisfied).

Because, 11.5.9 is NOT supported with JDK 5, I believe we are hitting the above issue (internal server error in installed base). As of now, there is no solution to this issue apart from
1. downgrading JDK from 5 to 1.4.2
2. modifying the xml to point to 1.4.2
3. running autoconfig to ensure the changes are reflected.

Signing off now hoping that the above solution works!

Find OS Kernel Bit in UNIX - Ready reckoner

Listed below is the ready reckoner for finding out the kernel bit information for Linux, Solaris, HP-UX and IBM AIX, which are the most commonly used OS for Oracle Apps.

Linux

getconf LONG_BIT
or
uname -m (works for x86-64 bit servers. Not tested it in 32 bit servers)

eg:
$ getconf LONG_BIT
64
$ uname -m
x86_64
$


Sun SPARC Solaris

isainfo -kv

eg:
$ isainfo -kv
64-bit sparcv9 kernel modules
$

HP-UX

getconf KERNEL_BITS

eg:
$ getconf KERNEL_BITS
64
$

IBM AIX

bootinfo -K

Not tested as I currently do not have any AIX servers to test on!


Note 1: To find whether a given executable is 32-bit or 64-bit, use the command

file <executable_name>
eg:
$ file oracle
oracle: ELF 64-bit MSB executable SPARCV9 Version 1, dynamically linked, not stripped
$ file tnslsnr
tnslsnr: ELF 64-bit MSB executable SPARCV9 Version 1, dynamically linked, not stripped
$ file sqlplus
sqlplus: ELF 64-bit MSB executable SPARCV9 Version 1, dynamically linked, not stripped


Note 2: You can install a 32-bit executable on a 64-bit OS but not vice-versa.

Monday, February 2, 2009

The curious case of database crash and ORA_NLS10

Case Facts:

Database Version ==> 10.2.0.3.0
Oracle Apps Version ==> 11.5.10.2
Node Type ==> Multi-node
RAC ==> No
OS & version ==> Sun SPARC Solaris (64-Bit) 10 (5.10)
Symptom ==> Database crashes with ORA-07445

Case Description:

While trying to bring up an Oracle Apps instance, I found that the database was crashing with ORA-07445 errors (as below) as soon as I brought up the concurrent managers.
_____________________________________________________________

Mon Feb 2 00:26:24 2009
Errors in file $ORACLE_HOME/admin/[CONTEXT_NAME]/udump/cpek_ora_6085.trc:
ORA-07445: exception encountered: core dump [lfilic()+328] [SIGSEGV] [Address not mapped to object] [0x000000040] [] []
ORA-29282: invalid file ID
Mon Feb 2 00:26:27 2009
Errors in file $ORACLE_HOME/admin/[CONTEXT_NAME]/udump/cpek_ora_5512.trc:
ORA-07445: exception encountered: core dump [lfilic()+328] [SIGSEGV] [Address not mapped to object] [0x000000040] [] []
Mon Feb 2 00:26:34 2009
Errors in file $ORACLE_HOME/admin/[CONTEXT_NAME]/udump/cpek_ora_5468.trc:
ORA-07445: exception encountered: core dump [lfilic()+328] [SIGSEGV] [Address not mapped to object] [0x000000040] [] []
Mon Feb 2 00:26:51 2009
Errors in file $ORACLE_HOME/admin/[CONTEXT_NAME]/udump/cpek_ora_5573.trc:
ORA-07445: exception encountered: core dump [lfilic()+328] [SIGSEGV] [Address not mapped to object] [0x000000040] [] []
Mon Feb 2 00:28:02 2009
Errors in file $ORACLE_HOME/admin/[CONTEXT_NAME]/udump/cpek_ora_5486.trc:
ORA-07445: exception encountered: core dump [lfilic()+328] [SIGSEGV] [Address not mapped to object] [0x000000040] [] []
Mon Feb 2 00:28:37 2009
.....
..
..
..
Mon Feb 2 00:38:10 2009
Errors in file $ORACLE_HOME/admin/[CONTEXT_NAME]/udump/cpek_ora_5560.trc:
ORA-07445: exception encountered: core dump [lfilic()+328] [SIGSEGV] [Address not mapped to object] [0x000000040] [] []
Mon Feb 2 00:40:15 2009
Incremental checkpoint up to RBA [0x27c.48028.0], current log tail at RBA [0x27d.1fc8.0]
Mon Feb 2 00:42:30 2009
Completed checkpoint up to RBA [0x27d.2.10], SCN: 10245291290327
Mon Feb 2 00:48:19 2009
MMNL: terminating instance due to error 472
Instance terminated by MMNL, pid = 4493
_____________________________________________________________

Suspects:

I found that this error is caused because the ORA_NLS10 variable is not set when the database was started or is set to an incorrect value.

Case Resolution:

1. set the env variable ORA_NLS10 as below:
export ORA_NLS10=$ORACLE_HOME/nls/data/9idata (for an Apps instance).

2. Startup the database.

3. Startup all the Apps services.

Witnesses:

1. Metalink Note: 420069.1

2. Metalink Note: 567472.1


AF Variables in CONTEXT_FILE - What they are and their use.

I was trying to find something in the CONTEXT_FILE when I happened to notice the AF_CLASSPATH env variable (not that I haven't noticed it earlier). But this time, it set me thinking as to what might its purpose be since we already had a CLASSPATH variable and its value seemed to be the same as the AF_CLASSPATH variable.

I found 3 variables starting with AF:
  1. AF_JRE_TOP
  2. AF_CLASSPATH
  3. AFJVAPRG.
I wanted to find out what they were and why they were used.

This is the explanation provided in Metalink Note 412709.1 (Oracle Workflow Documentation Updates for 11i.ATG_PF.H.delta.5 (RUP 5))

AFJVAPRG - The location of the JDK or JRE executable for the concurrent processing tier.

AF_CLASSPATH - The classpath for the concurrent processing tier.

Also, Note 373386.1 has this to say about AF_CLASSPATH.

The AF_CLASSPATH variable is used by JAVA concurrent programs and must use literal (full) path values.


By combining the above two notes, we can perhaps safely say that the AF variables are set for letting programs know the location of java (AFJVAPRG), JRE (AF_JRE_TOP) and the classpath (AF_CLASSPATH) on the concurrent node.

Also, note that these variables are also set on the web node (and must be set to valid values too).