Monday, February 2, 2009

The curious case of database crash and ORA_NLS10

Case Facts:

Database Version ==> 10.2.0.3.0
Oracle Apps Version ==> 11.5.10.2
Node Type ==> Multi-node
RAC ==> No
OS & version ==> Sun SPARC Solaris (64-Bit) 10 (5.10)
Symptom ==> Database crashes with ORA-07445

Case Description:

While trying to bring up an Oracle Apps instance, I found that the database was crashing with ORA-07445 errors (as below) as soon as I brought up the concurrent managers.
_____________________________________________________________

Mon Feb 2 00:26:24 2009
Errors in file $ORACLE_HOME/admin/[CONTEXT_NAME]/udump/cpek_ora_6085.trc:
ORA-07445: exception encountered: core dump [lfilic()+328] [SIGSEGV] [Address not mapped to object] [0x000000040] [] []
ORA-29282: invalid file ID
Mon Feb 2 00:26:27 2009
Errors in file $ORACLE_HOME/admin/[CONTEXT_NAME]/udump/cpek_ora_5512.trc:
ORA-07445: exception encountered: core dump [lfilic()+328] [SIGSEGV] [Address not mapped to object] [0x000000040] [] []
Mon Feb 2 00:26:34 2009
Errors in file $ORACLE_HOME/admin/[CONTEXT_NAME]/udump/cpek_ora_5468.trc:
ORA-07445: exception encountered: core dump [lfilic()+328] [SIGSEGV] [Address not mapped to object] [0x000000040] [] []
Mon Feb 2 00:26:51 2009
Errors in file $ORACLE_HOME/admin/[CONTEXT_NAME]/udump/cpek_ora_5573.trc:
ORA-07445: exception encountered: core dump [lfilic()+328] [SIGSEGV] [Address not mapped to object] [0x000000040] [] []
Mon Feb 2 00:28:02 2009
Errors in file $ORACLE_HOME/admin/[CONTEXT_NAME]/udump/cpek_ora_5486.trc:
ORA-07445: exception encountered: core dump [lfilic()+328] [SIGSEGV] [Address not mapped to object] [0x000000040] [] []
Mon Feb 2 00:28:37 2009
.....
..
..
..
Mon Feb 2 00:38:10 2009
Errors in file $ORACLE_HOME/admin/[CONTEXT_NAME]/udump/cpek_ora_5560.trc:
ORA-07445: exception encountered: core dump [lfilic()+328] [SIGSEGV] [Address not mapped to object] [0x000000040] [] []
Mon Feb 2 00:40:15 2009
Incremental checkpoint up to RBA [0x27c.48028.0], current log tail at RBA [0x27d.1fc8.0]
Mon Feb 2 00:42:30 2009
Completed checkpoint up to RBA [0x27d.2.10], SCN: 10245291290327
Mon Feb 2 00:48:19 2009
MMNL: terminating instance due to error 472
Instance terminated by MMNL, pid = 4493
_____________________________________________________________

Suspects:

I found that this error is caused because the ORA_NLS10 variable is not set when the database was started or is set to an incorrect value.

Case Resolution:

1. set the env variable ORA_NLS10 as below:
export ORA_NLS10=$ORACLE_HOME/nls/data/9idata (for an Apps instance).

2. Startup the database.

3. Startup all the Apps services.

Witnesses:

1. Metalink Note: 420069.1

2. Metalink Note: 567472.1


1 comment:

Anonymous said...

Wow Praveen,

I had exactly the same issue, and your solution has helped me a lot

A.Kishore

http://www.appsdba.info