Monday, March 30, 2015

Set Hugepages for Oracle 11gR2 database

1. Execute the shell script provided in Note ID: 401749.1 to find out the ideal setting for the parameter vm.nr_hugepages, set it in /etc/sysctl.conf and reboot VM/Server.

Script from Note 401749.1

-bash-3.2$ cat huge_pages_calc.sh
#!/bin/bash
#
# hugepages_settings.sh
#
# Linux bash script to compute values for the
# recommended HugePages/HugeTLB configuration
#
# Note: This script does calculation for all shared memory
# segments available when the script is run, no matter it
# is an Oracle RDBMS shared memory segment or not.
#
# This script is provided by Doc ID 401749.1 from My Oracle Support
# http://support.oracle.com

# Welcome text
echo "
This script is provided by Doc ID 401749.1 from My Oracle Support
(http://support.oracle.com) where it is intended to compute values for
the recommended HugePages/HugeTLB configuration for the current shared
memory segments. Before proceeding with the execution please note following:
 * For ASM instance, it needs to configure ASMM instead of AMM.
 * The 'pga_aggregate_target' is outside the SGA and
   you should accommodate this while calculating SGA size.
 * In case you changes the DB SGA size,
   as the new SGA will not fit in the previous HugePages configuration,
   it had better disable the whole HugePages,
   start the DB with new SGA size and run the script again.
And make sure that:
 * Oracle Database instance(s) are up and running
 * Oracle Database 11g Automatic Memory Management (AMM) is not setup
   (See Doc ID 749851.1)
 * The shared memory segments can be listed by command:
     # ipcs -m


Press Enter to proceed..."

read

# Check for the kernel version
KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'`

# Find out the HugePage size
HPG_SZ=`grep Hugepagesize /proc/meminfo | awk '{print $2}'`
if [ -z "$HPG_SZ" ];then
    echo "The hugepages may not be supported in the system where the script is being executed."
    exit 1
fi

# Initialize the counter
NUM_PG=0

# Cumulative number of pages required to handle the running shared memory segments
for SEG_BYTES in `ipcs -m | cut -c44-300 | awk '{print $1}' | grep "[0-9][0-9]*"`
do
    MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`
    if [ $MIN_PG -gt 0 ]; then
        NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`
    fi
done

RES_BYTES=`echo "$NUM_PG * $HPG_SZ * 1024" | bc -q`

# An SGA less than 100MB does not make sense
# Bail out if that is the case
if [ $RES_BYTES -lt 100000000 ]; then
    echo "***********"
    echo "** ERROR **"
    echo "***********"
    echo "Sorry! There are not enough total of shared memory segments allocated for
HugePages configuration. HugePages can only be used for shared memory segments
that you can list by command:

    # ipcs -m

of a size that can match an Oracle Database SGA. Please make sure that:
 * Oracle Database instance is up and running
 * Oracle Database 11g Automatic Memory Management (AMM) is not configured"
    exit 1
fi

# Finish with results
case $KERN in
    '2.2') echo "Kernel version $KERN is not supported. Exiting." ;;
    '2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;
           echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;
    '2.6') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    '3.8') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
esac

# End


-bash-3.2$ ./huge_pages_calc.sh

This script is provided by Doc ID 401749.1 from My Oracle Support
(http://support.oracle.com) where it is intended to compute values for
the recommended HugePages/HugeTLB configuration for the current shared
memory segments. Before proceeding with the execution please note following:
 * For ASM instance, it needs to configure ASMM instead of AMM.
 * The 'pga_aggregate_target' is outside the SGA and
   you should accommodate this while calculating SGA size.
 * In case you changes the DB SGA size,
   as the new SGA will not fit in the previous HugePages configuration,
   it had better disable the whole HugePages,
   start the DB with new SGA size and run the script again.
And make sure that:
 * Oracle Database instance(s) are up and running
 * Oracle Database 11g Automatic Memory Management (AMM) is not setup
   (See Doc ID 749851.1)
 * The shared memory segments can be listed by command:
     # ipcs -m


Press Enter to proceed...

Recommended setting: vm.nr_hugepages = 3076


2. Set memlock limit at OS level.

a. Modify /etc/security/limits.conf and enter the below values:

oracle  soft    memlock
oracle  hard    memlock

memlock_value = (Huge_Pages * Huge_Page_Size)

Huge_Pages (value obtained by script in point 1):

or

sysctl -a | grep -i vm.nr_hugepages

Huge_Page_Size:

grep Hugepagesize /proc/meminfo


Note: You can check the current value or memlock using ulimit -l.


3. Set kernel parameters shmmax and shmall.

kernel.shmmax = (1/2 of RAM) and sga_max_size < kernel.shmmax

So, if sga_max_size is ½ or RAM, then set shmmax to around 60 or 70% of RAM.

kernel.shmall = (SGA_MAX_SIZE + PGA_AGGREGATE_TARGET)/PAGE_SIZE

PAGE_SIZE_VALUE:

$ getconf PAGE_SIZE


4. Disable Automatic Memory Management (AMM) if necesary as it is incompatible with HugePages.

Unset the init parameter memory_target parameter, if set.


5. After the reboot, start up the DB and check the below section in the alert log:

****************** Large Pages Information *****************

Total Shared Global Region in Large Pages = 0 KB (0%)

Large Pages used by this instance: 0 (0 KB)
Large Pages unused system wide = 10000 (20 GB) (alloc incr 64 MB)
Large Pages configured system wide = 10000 (20 GB)
Large Page size = 2048 KB

RECOMMENDATION:
  Total Shared Global Region size is 24 GB. For optimal performance,
  prior to the next instance restart increase the number
  of unused Large Pages by atleast 2292 2048 KB Large Pages (4584 MB)
  system wide to get 100% of the Shared
  Global Region allocated with Large pages
***********************************************************

Successful message would be as below:

****************** Large Pages Information *****************

Total Shared Global Region in Large Pages = 62 GB (100%)

Large Pages used by this instance: 31745 (62 GB)
Large Pages unused system wide = 1023 (2046 MB) (alloc incr 128 MB)
Large Pages configured system wide = 32768 (64 GB)
Large Page size = 2048 KB
***********************************************************


Notes:

1. USE_LARGE_PAGES è This parameter is new in 11gR2 (11.2.0.2) onwards. It is set to true by default.

In Oracle Database 11g Release 2 (11.2.0.2), if there are not enough large pages configured on the system, then regular sized pages will be used to allocate SGA memory. This can cause the free large pages to go unused, and the operating system can allocate a huge amount of memory to create page tables to map SGA into physical pages for the Oracle processes. This may lead to ORA-04030 errors and severe performance degradation on an instance.

In Oracle Database 11g Release 2 (11.2.0.3), Oracle allocates as much of the SGA as it can in large pages, and if it runs out, it will allocate the rest of the SGA using regular sized pages. This can cause the instance to create additional shared memory segments for the SGA, but the total SGA size will be unchanged. In this supported mixed page mode allocation, the database will exhaust the available large pages before switching to regular sized pages.


References:


2. 361323.1 - HugePages on Linux: What It Is... and What It Is Not...

3. 401749.1 - Shell Script to Calculate Values Recommended Linux HugePages / HugeTLB Configuration

4. 361468.1 - HugePages on Oracle Linux 64-bit



7. http://www.oracle-base.com/articles/linux/configuring-huge-pages-for-oracle-on-linux-64.php

8. 1392543.1 - Large Pages Information in the Alert Log

9. 1392497.1 - USE_LARGE_PAGES To Enable HugePages In 11.2

10. Bug 9195408 - DB STARTUP DOES NOT CHECK WHETHER HUGEPAGES ARE ALLOCATED- PROVIDE USE_HUGEPAGES

11. http://agorbyk.wordpress.com/2012/02/19/oracle-11-2-0-3-and-hugepages-allocation/

12. http://kevinclosson.wordpress.com/category/use_large_pages/


14. 803238.1 - Oracle Not Utilizing Hugepages