Oracle 11gR2 RAC on HPUX Itanium 11.31

This was my RAC installation on around september last year on HPUX. 
Here I am sharing my action plan as it is I prepared on notepad.

Oracle Version : 11.2.0.3
OS : HPUX Itanium 11.31

1. configure NTP

2. configure DNS using SAM

3 . Create OS groups using the command below. Enter these commands as the 'root' user:
/usr/sbin/groupadd -g 503 oinstall
/usr/sbin/groupadd -g 504 dba
/usr/sbin/groupadd -g 505 asmadmin
/usr/sbin/groupadd -g 506 asmdba
/usr/sbin/groupadd -g 507 asmoper

4. Create the users that will own the Oracle software using the commands:

*** please use "/usr/bin/ksh" shell for oracle user
#/usr/sbin/useradd -u 503 -s /usr/bin/ksh -g oinstall -G dba,asmadmin,asmdba,asmoper -p /home/oracle oracle

5. Set the password for the oracle account. [Oracle6!7]

6. Sample /etc/hosts file will be as below: "**also configure DNS with reverse resolve":
127.0.0.1 localhost.localdomain localhost #dont add anyhing extra in this line
#lan1 - PUBLIC
10.101.4.57  ORAdb1.robi.com.bd ORAdb1
10.101.4.58  ORAdb2.robi.com.bd ORAdb2
#VIP
10.101.4.59  ORAdb1-vip.robi.com.bd ORAdb1-vip
10.101.4.60  ORAdb2-vip.robi.com.bd ORAdb2-vip
#lan3 - PRIVATE
172.16.4.57 ORAdb1-pvt
172.16.4.58 ORAdb2-pvt


7. Kernel parameters: (can also be fixed during installation using runfixup.sh)
ksi_alloc_max 32768
executable_stack 0
ksi_alloc_max 32768
max_thread_proc 1024
maxdsiz 1073741824 (1 GB)
maxdsiz_64bit 2147483648 (2 GB)
maxfiles 16384
maxfiles_lim 65536
maxssiz 134217728 (128 MB)
maxssiz_64bit 1073741824 (1 GB)
maxuprc 3686
msgmni 4096
msgtql 4096
ncsize 35840
nflocks 4096
ninode 34816
nkthread 7184
nproc 4096
semmni 4096
semmns 8192
semmnu 4096
semvmx 32767
shmmax 1073741824
shmmni 4096
shmseg 512
tcp_smallest_anon_port 9000
tcp_largest_anon_port 65500
udp_smallest_anon_port 9000
udp_largest_anon_port 65500


8. install below packages or upper versions:
Operating system: 
  HP-UX 11iV3 patch Bundle Sep/ 2008 (B.11.31.0809.326a)
HP-UX 11.31 packages and bundles:
  PHCO_41479 (or later) 11.31 diskowner(1M) cumulative patch
  PHKL_38038 vm cumulative patch
  PHKL_38938 11.31 SCSI cumulative I/O patch 
  PHKL_40941  Scheduler patch : post wait hang 
  PHSS_36354 11.31 assembler patch
  PHSS_37042  11.31 hppac (packed decimal)
  PHSS_37959  Libcl patch for alternate stack issue fix (QXCR1000818011)
  PHSS_39094 11.31 linker + fdp cumulative patch
  PHSS_39100 11.31 Math Library Cumulative Patch
  PHSS_39102 11.31 Integrity Unwind Library
  PHSS_38141 11.31 aC++ Runtime
  
To avoid the node evictions:  
  PHKL_40208
  PHKL_40372
  
HP-UX Logical Volume Manager (LVM): 
  PHCO_41479 (or later) 11.31 
C / C++ compiler: 
  HP C/aC++ A.06.20 (Swlist Bundle - C.11.31.04) - September 2008
C Compiler Patches:
  PHSS_39824 11.31 HP C/aC++ Compiler (A.06.23)
  PHSS_39826 11.31 u2comp/be/plug-in (C.06.23)
Gcc Compiler:
  Gcc 4.2.3
  X11MotifDevKit.MOTIF21-PRG
Oracle JDBC/OCI Drivers:
HPUX JDK 6.0.05
  HPUX JDK 5.0.15
Oracle ODBC:
  You can download and install the Driver Manager from the following URL:http://www.unixodbc.org
IBM MQ Series V. 6.0, client and server (optional):
  MQSERIES.MQM-CL-HPUX
  MQSERIES.MQM-SERVER
Motif 2.1 Development Environment package (optional):  
  TIBCO Rendezvous 7.2  
Perl:
Perl 5.8.8
Programming Software (optional):
  Pro*Cobol - Micro Focus Server Express 5.1
  Pro*FORTRAN - HP FORTRAN/90 - Sep 2008 release
SSH: 
  The required SSH software is the default SSH shipped with your operating system.


10 Creating Required Symbolic Links
This task is required only if the Motif 2.1 Development Environment package (X11MotifDevKit.MOTIF21-PRG) is not installed.
# cd /usr/lib
ln -s libX11.3 libX11.sl
ln -s libXIE.2 libXIE.sl
ln -s libXext.3 libXext.sl
ln -s libXhp11.3 libXhp11.sl
ln -s libXi.3 libXi.sl
ln -s libXm.4 libXm.sl
ln -s libXp.2 libXp.sl
ln -s libXt.3 libXt.sl
ln -s libXtst.2 libXtst.sl

11. Create the Oracle Inventory Directory
mkdir -p /u01/app/oraInventory
chown -R oracle:oinstall /u01/app/oraInventory
chmod -R 775 /u01/app/oraInventory

12. Creating the Oracle Grid Infrastructure Home Directory
mkdir -p /u01/app/11.2.0/grid
chown -R oracle:oinstall /u01/app/11.2.0/grid
chmod -R 775 /u01/app/11.2.0/grid

vi oracrs.sh
export ORACLE_SID=+ASM1/+ASM2
export ORACLE_HOME=/u01/app/11.2.0/grid
export PATH=$ORACLE_HOME/bin:$PATH

13. Creating the Oracle Base Directory
mkdir -p /u01/app/oracle/product/11.2.0/dbhome_1
chown -R oracle:oinstall /u01/app/oracle/
chmod -R 775 /u01/app

vi oradb.sh
export ORACLE_SID=ORADB1/ORADB2
export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
export PATH=$ORACLE_HOME/bin:$PATH

su - oracle
vi .profile and append below on both node:
export ORACLE_HOSTNAME=ORAdb1.robi.com.bd

14. ASM DIsk Group sizing
DATA01     200G
SYSTEMDG   100G
RECO       100G


15. Configuring Disk Devices for Oracle ASM    
    
    a. To make sure that the disks are available, enter the following command on every node:
       # ioscan -fNnC disk
              
    b. If the ioscan command does not display device name information for a device, enter the following command to install the special device files for any new devices:

       # /usr/sbin/insf -e
    c. For each disk to add to a disk group, enter the following command on any node to verify that it is not already part of an LVM volume group:

       # /sbin/pvdisplay /dev/dsk/cxtydz

       If this command displays volume group information, the disk is already part of a volume group. 
       The disks that you choose must not be part of an LVM volume group.

    d. The disk names for the same disk can be different on each node. A disk can be identified as the same
      one via the WWID. The WWID of a disk can be checked via the following command:
        # scsimgr lun_map -D /dev/rdisk/disk29 | grep WWID

        ORADB1:/home/root #scsimgr lun_map -D /dev/rdisk/disk29 | grep WWID
        World Wide Identifier(WWID)    = 0x6006016001102d00b81efc7444e1e111
        ORADB2:/home/root #scsimgr lun_map -D /dev/rdisk/disk30 | grep WWID
        World Wide Identifier(WWID)    = 0x6006016001102d00b81efc7444e1e111

        ORADB1:/home/root #scsimgr lun_map -D /dev/rdisk/disk30 | grep WWID
        World Wide Identifier(WWID)    = 0x6006016001102d00b21efc7444e1e111
        ORADB2:/home/root #scsimgr lun_map -D /dev/rdisk/disk29 | grep WWID
        World Wide Identifier(WWID)    = 0x6006016001102d00b21efc7444e1e111

        ORADB1:/home/root #scsimgr lun_map -D /dev/rdisk/disk31 | grep WWID
        World Wide Identifier(WWID)    = 0x6006016001102d00b61efc7444e1e111
        ORADB2:/home/root #scsimgr lun_map -D /dev/rdisk/disk31 | grep WWID
        World Wide Identifier(WWID)    = 0x6006016001102d00b61efc7444e1e111

        ORADB1:/home/root #scsimgr lun_map -D /dev/rdisk/disk32 | grep WWID
        World Wide Identifier(WWID)    = 0x6006016001102d00ba1efc7444e1e111
        ORADB2:/home/root #scsimgr lun_map -D /dev/rdisk/disk32 | grep WWID
        World Wide Identifier(WWID)    = 0x6006016001102d00ba1efc7444e1e111

        
    e. create a special Oracle device directory and to use mknod to create device paths in this special Oracle folder
       # mkdir /dev/oracle
       # ll /dev/rdisk/disk29
       crw-r----- 1 bin sys 23 0x000019 Jan 16 12:16 /dev/dsk/cxtydz

       # mknod /dev/oracle/asmdisk1 c 23 0x000019
        
        on ORAdb1
        ---------
        #ll /dev/rdisk/disk29
        #mknod /dev/oracle/asm_systemdg_01 c 13 0x000013
        #ll /dev/rdisk/disk30
        #mknod /dev/oracle/asm_data01_01 c 13 0x000014
        #ll /dev/rdisk/disk31
        #mknod /dev/oracle/asm_data01_02 c 13 0x000015
        #ll /dev/rdisk/disk32
        #mknod /dev/oracle/asm_reco_01 c 13 0x000016

       Later during set-up, for the ASM instance, set the ASM_DISKSTRING parameter to /dev/oracle/*  #*/
    f. Modify the owner, group, and permissions on the character raw device files (ASM and Voting Disks) on all nodes:
       # chown -R oracle:asmadmin /dev/oracle                      #*/
       # chmod -R 660 /dev/oracle                                  #*/
  

16. Setting the Minor Number for Device Files

    a. Log in as the root user.

    b.Determine whether /dev/async exists. If the device does not exist, then use the following command to create it:

    # /sbin/mknod /dev/async c 101 0x4

    c. Alternatively, you can set the minor number value to 0x104 using the following command:

    # /sbin/mknod /dev/async c 101 0x104

    d. If /dev/async exists, then determine the current value of the minor number, as shown in the following example:

       #ll  -l /dev/async
       crw-rw-rw-   1 bin        bin        101 0x000000 Jul 26 15:45 /dev/async

    e. If the existing minor number of the file is not 0x4 or 0x104, then change it to an expected value using one of the following commands:
    # rm /dev/async
    # mknod /dev/async c 101 0x4
    or
    # mknod /dev/async c 101 0x104

       
17. Oracle Grid Infrastructure Install
a. As the grid user (Grid Infrastructure software owner) start the installer by running "runInstaller" from the staged installation media
b. Choose Skip Software Updates
c. Select radio button 'Install and Configure Grid Infrastructure for a Cluster' and click ' Next> '
d. Select radio button 'Advanced Installation' and click ' Next> 
e. Accept 'English' as language' and click  Next> 
f. Specify your cluster name and the SCAN name you want to use and click  Next> 
g. Uncheck Configure GNS
h. Use the Edit and Add buttons to specify the node names and virtual IP addresses you configured previously in your /etc/hosts file. 
   Use the 'SSH Connectivity' button to configure/test the passwordless SSH connectivity between your nodes.
   *** some times need to configure ssh manualy
i. Type in the OS password for the user 'oracle' and press 'Setup' and test and ok 'Next>'
j. Click on 'Interface Type' next to the Interfaces you want to use for your cluster and select the correct values for 'Public', 'Private' and 'Do Not Use' 
   When finished click ' Next> ' [we can use multiple NIC as private or public]
k. Select radio button Automatic Storage Management (ASM) and click ' Next> '
l. Select the 'DiskGroup Name' specify the 'Redundancy' and tick the disks you want to use,[if candidate disks are not shown then check permission for the /dev... folders] when done click ' Next> '
  [if no candidate disks found, then change discovery path to /dev/oracleasm/disks* or ORCL:*]
  NOTE: The number of voting disks that will be created depend on the redundancy level you specify:
  EXTERNAL will create 1 voting disk, NORMAL will create 3 voting disks, HIGH will create 5 voting disks   
  *** if disk are ok on both node and it is sure that "/dev/oracle/.." names are pointing to the same disk device hen ignore "PRVF-5150" error
m. Specify and conform the password you want to use and click ' Next> '
n. Select NOT to use IPMI and click ' Next> '
o. Assign groups as below:
   Oracle ASM DBA: asmdba
   Oracle ASM Operator: asmoper
   Oracle ASM Administrator: asmadmin
p. Specify the locations for your ORACLE_BASE and for the Software location and click ' Next> '
q. Specify the locations for your Inventory directory and click ' Next>'
r. Check that status of all checks is Succeeded and click ' Next> '
s. Wait for the OUI to complete its tasks
t. At this point you may need to run oraInstRoot.sh on all cluster nodes (if this is the first installation of an Oracle product on this system)
   Run them on the first node alone then go for other nodes
   Then press ok
   NOTE: root.sh should be run on one node at a time.
   if root.sh fails, then resolve it and run again
u. Wait for the OUI to finish the cluster configuration.

check OCR,Voting & OLR :
ocrcheck
crsctl query css votedisk
grid_home/cdata/.olr 


18. RDBMS Software Install   

a. #su - oracle
change into the directory where you staged the RDBMS software
./runInstaller

b. Provide your e-mail address, tick the check box and provide your Oracle Support Password 
   if you want to receive Security Updates from Oracle Support and click ' Next> ' 
c. Select the option 'Install Database software only' and click ' Next> '
d. Select 'Real Application Clusters database installation', and select all nodes. 
   Use the 'SSH Connectivity' button to configure/test the passwordless SSH connectivity between your nodes 
e. Type in the OS password for the oracle user and click 'Setup'   
f. To confirm English as selected language click ' Next> '
g. Make sure radio button 'Enterprise Edition'/'Standard Edition' is ticked, click ' Next> '
h. Specify path to your Oracle Base and below to the location where you want to store the software (Oraclehome). Click ' Next> '
i. Specify groups:
   Database Administrator: dba
   Database Operator:oinstall
j. Oracle Universal Installer performs prerequisite checks.
k. Check that the status of all checks is 'Succeeded' and click ' Next> '
l. Log in to a terminal window as root user and run the root.sh script on the first node. 
   When finished do the same for all other nodes in your cluster as well. When finished click 'OK'
   NOTE: root.sh should be run on one node at a time.
m. Click ' Close ' to finish the installation of the RDBMS Software.

19. Run ASMCA to create diskgroups

a. #su - grid
   cd /u01/11.2.0/grid/bin
   ./asmca
b. Click 'Create' to create a new diskgroup
c. Type in a name for the diskgroup, select the redundancy you want to provide and mark the tick box for the disks you want to assign to the new diskgroup   
d. Click 'OK'
e. Click 'Create' to create the diskgroup for the flash recovery area
f. Type in a name for the diskgroup, select the redundancy you want to provide and mark the tick box for the disks you want to assign to the new diskgroup.
g. Click 'OK'
h. Click 'Exit'


20. It is Oracles Best Practice to have an OCR mirror stored in a second diskgroup. 
    To follow this recommendation add an OCR mirror. Mind that you can only have one OCR in a diskgroup.
Action:
a. To add OCR mirror to an Oracle ASM diskgroup, ensure that the Oracle Clusterware stack is running and
run the following command as root from the GridInfrastructureHome? /bin directory:
b. # ocrconfig -add +DATA1
c. # ocrcheck

21. Run DBCA to create the database
a. #su - oracle
   cd /u01/app/oracle/product/11.2.0/db_1/bin
   ./dbca
b. Select 'Oracle Real Application Clusters database' and click 'Next'
c. choose option 'Create a Database' and click 'Next'
d. Select the database template that you want to use for your database and click 'Next'
e. Type in the name you want to use for your database and select all nodes before you click 'Next'
f. select the options you want to use to manage your database and click 'Next'
g. Type in the passwords you want to use and click 'Next'
h. Select the diskgroup you created for the database files and click 'Multiplex Redo Logs and Control Files'. 
   In the popup window define the diskgroup that should contain controlfiles and 
   redo logfiles and the diskgroup that should contain the mirrored files.   
   When all file destinations are correct click 'Next'
i. Specify the diskgroup that was created for the flash recovery area and define the size. 
   If the size is smaller than recommended a warning will popup.   
j. Select if you want to have sample schemas created in your database and click 'Next'
k. Review and change the settings for memory allocation, characterset etc. according to your needs and click 'Next'
l. Review the database storage settings and click 'Next'
m. Make sure the tickbox 'Create Database' is ticked and click 'Finish'
n. Review the database configuration details again and click 'OK'
o. The database is now created, you can either change or unlock your passwords or just click Exit to finish the database creation

22. For all the luns added in ASM, set LV timeout value = (# of paths * PV Timeout) + 10 seconds, using "lvchange -t timeinsec /dev/dsk/cxtydz" command
       in our case, EMC recommended PV timeout = 90
                                    # of path  = 4
       then check with # lvdisplay -v /dev/dsk/cxtydz


Sample change cluster resource
------------------------
ex: change listener port
------------------------
emca -deconfig dbcontrol db -repos drop -cluster
- srvctl modify listener -l -o -p
- restart the full cluster [ crsctl stop/start crs]
- check with "lsnrctl status", from both DB & AMS "show parameter local"
emca -config dbcontrol db -repos create -cluster


-----------------------------
ex: change scan listener port
-----------------------------
emca -deconfig dbcontrol db -repos drop -cluster
- srvctl modify scan_listener -p 1521
- alter system set remote_listener='ORAdb-scan.robi.com.bd:1521' scope=both;--from DB & ASM on all the db nodes
- Change the necessary files in $ORACLE_HOME/network/admin on server & client
- restart the full cluster [ crsctl stop/start crs]
- check with "lsnrctl status", from both DB & AMS "show parameter remote"
- from all DB nodes "lsnrctl status LISTENER", also check same for all scan lsitener
emca -config dbcontrol db -repos create -cluster
---------------
emca -deconfig dbcontrol db -repos drop -cluster
emca -config dbcontrol db -repos create -cluster



http://docs.oracle.com/cd/E11882_01/install.112/e10815/typinstl.htm#BABFEGHH
http://docs.oracle.com/cd/E11882_01/install.112/e10815/pre_hpux.htm#CIHFICFD
http://docs.oracle.com/cd/E11882_01/install.112/e10815/manpreins.htm#BABCHAED
http://docs.oracle.com/cd/E11882_01/install.112/e10815/pre_hpux.htm#CEGGJDIG
http://docs.oracle.com/cd/E11882_01/install.112/e10815/storage.htm#CDEECFGI
http://gjilevski.wordpress.com/2010/10/03/fresh-oracle-11-2-0-2-grid-infrastructure-installation-prvf-5150-prvf-5184/

MOS Doc: RAC and Oracle Clusterware Best Practices and Starter Kit (Platform Independent) [ID 810394.1]

Install Oracle RAC 11gR2 on Redhat Linux

While installing an Oracle RAC on around March last year, I prepared below steps on notepad to follow during installation.
I am sharing this as it is and hope it will help others.

Oracle Version : 11.2.0.3
OS : RHEL 5.5 64 Bit 

1. Network Check List:
- The SCAN addresses need to be on the same subnet as the VIP addresses for nodes in the cluster
- Each node must have at least two network interface cards (NIC), or network adapters
- Public interface names must be the same for all nodes. If the public interface on one node uses the network adapter eth0, then you must configure eth0 as the public interface on all nodes
- Private interface names should be the same for all nodes as well. If eth1 is the private interface name for the first node, then eth1 should be the private interface name for your second node
- The network adapter for the private interface must support the user datagram protocol (UDP) using high-speed network adapters and a network switch that supports TCP/IP (Gigabit Ethernet or better)
- The host name of each node must conform to the RFC 952 standard, which permits alphanumeric characters. Host names using underscores ("_") are not allowed

2. IP Address Requirements:
- One public IP address for each node
- One virtual IP address for each node
- Three single client access name (SCAN) addresses for the cluster
- 1 private IP address for each machine

3. Prepare the cluster nodes for Oracle RAC:
NOTE: We recommend different users for the installation of the Grid Infrastructure (GI) and the Oracle
RDBMS home. The GI will be installed in a separate Oracle base, owned by user 'grid.' After the grid install
the GI home will be owned by root, and inaccessible to unauthorized users.

# configure NTP
----------------
# configure DNS in /etc/resolve.conf
------------------------------------

a. Create OS groups using the command below. Enter these commands as the 'root' user:
/usr/sbin/groupadd -g 503 oinstall
/usr/sbin/groupadd -g 504 dba
/usr/sbin/groupadd -g 505 asmadmin
/usr/sbin/groupadd -g 506 asmdba
/usr/sbin/groupadd -g 507 asmoper

b. Create the users that will own the Oracle software using the commands:

#/usr/sbin/useradd -u 503 -g oinstall -G dba,asmadmin,asmdba,asmoper oracle

c. Set the password for the oracle account using the following command. Replace password with your own password

4. Networking
a. Determine your cluster name.
b. Determine the public host name for each node in the cluster. For the public host name, use the primary host name of each node. 
   In other words, use the name displayed by the hostname command for example: racnode1
c. Determine the public virtual hostname for each node in the cluster. The virtual host name is a public node
   name that is used to reroute client requests sent to the node if the node is down. Oracle recommends that you
   provide a name in the format -vip, for example: racnode1-vip. The virutal hostname must
   meet the following requirements:
  - The virtual IP address and the network name must not be currently in use.
  - The virtual IP address must be on the same subnet as your public IP address.
  - The virtual host name for each node should be registered with your DNS.   
d. Determine the private hostname for each node in the cluster. This private hostname does not need to be
   resolvable through DNS and should be entered in the /etc/hosts file. A common naming convention for the
   private hostname is -pvt.
  - The private IP should NOT be accessable to servers not participating in the local cluster.
  - The private network should be on standalone dedicated switch(es).
  - The private network should NOT be part of a larger overall network topology.
  - The private network should be deployed on Gigabit Ethernet or better.
It is recommended that redundant NICs are configured with the Linux bonding driver.
Active/passive is the preferred bonding method due to its simplistic configuration.

e. Define a SCAN DNS name for the cluster that resolves to three IP addresses (round-robin). 
  SCAN IPs must NOT be in the /etc/hosts file, the SCAN name must be resolved by DNS.

f. Sample /etc/hosts file will be as below:
127.0.0.1 localhost.localdomain localhost #dont add anyhing extra in this line
#eth0 - PUBLIC
10.101.5.25  oradb1.robi.com.bd oradb1
10.101.5.26  oradb2.robi.com.bd oradb2
#VIP
10.101.5.24  oradb1-vip.robi.com.bd oradb1-vip
10.101.5.31  oradb2-vip.robi.com.bd oradb2-vip
#eth1 - PRIVATE
172.16.5.25 oradb1-pvt
172.16.5.26 oradb2-pvt  


5. Configuring Kernel Parameters 
#vi /etc/sysctl.conf

kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 6553600
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
NOTE: The latest information on kernel parameter settings for Linux can be found in My Oracle Support ExtNote:169706.1

#/sbin/sysctl -p

6. Set shell limits for the oracle user
a. Add the following lines to the /etc/security/limits.conf file:
oracle               soft    nproc   2047
oracle               hard    nproc   16384
oracle               soft    nofile  1024
oracle               hard    nofile  65536

b. Add or edit the following line in the /etc/pam.d/login file, if it does not already exist:
session required pam_limits.so

c. Make the following changes to the default shell startup file, add the following lines to the /etc/profile file:
if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi

7. Create the Oracle Inventory Directory
# mkdir -p /oracle/oraInventory
# chown -R oracle:oinstall /oracle/oraInventory
# chmod -R 775 /oracle/orabase/oraInventory

8. Creating the Oracle Grid Infrastructure Home Directory
# mkdir -p /oracle/11.2.0/grid
# chown -R oracle:oinstall /oracle/11.2.0/grid
# chmod -R 775 /oracle/11.2.0/grid

vi oracrs.sh
export ORACLE_SID=+ASM1/+ASM2
export ORACLE_HOME=/u01/app/11.2.0/grid
export PATH=$ORACLE_HOME/bin:$PATH

9. Creating the Oracle Base Directory
# mkdir -p /oracle/orabase/product/11.2.0/dbhome_1
# chown -R oracle:oinstall /oracle/orabase
# chmod -R 775 /oracle/orabase

vi oradb.sh
export ORACLE_SID=ORADB1/ORADB2
export ORACLE_HOME=/oracle/orabase/product/11.2.0/dbhome_1
export PATH=$ORACLE_HOME/bin:$PATH



10. Check OS Software Requirements
binutils-2.15.92.0.2
compat-libstdc++-33-3.2.3
compat-libstdc++-33-3.2.3 (32 bit)
elfutils-libelf-0.97
elfutils-libelf-devel-0.97
expat-1.95.7
gcc-3.4.6
gcc-c++-3.4.6
glibc-2.3.4-2.41
glibc-2.3.4-2.41 (32 bit)
glibc-common-2.3.4
glibc-devel-2.3.4
glibc-headers-2.3.4
libaio-0.3.105
libaio-0.3.105 (32 bit)
libaio-devel-0.3.105
libaio-devel-0.3.105 (32 bit)
libgcc-3.4.6
libgcc-3.4.6 (32-bit)
libstdc++-3.4.6
libstdc++-3.4.6 (32 bit)
libstdc++-devel 3.4.6
make-3.80
pdksh-5.2.14
sysstat-5.0.5
unixODBC-2.2.11
unixODBC-2.2.11 (32 bit)
unixODBC-devel-2.2.11
unixODBC-devel-2.2.11 (32 bit)

The following command can be run on the system to list the currently installed packages:
rpm -q --qf '%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n' binutils \
compat-libstdc++-33 \
elfutils-libelf \
elfutils-libelf-devel \
gcc \
gcc-c++ \
glibc \
glibc-common \
glibc-devel \
glibc-headers \
ksh \
libaio \
libaio-devel \
libgcc \
libstdc++ \
libstdc++-devel \
make \
sysstat \
unixODBC \
unixODBC-devel

NOTE: Be sure to check on all nodes that the Linux Firewall and SE Linux is disabled.

11. Prepare the shared storage for Oracle RAC
- All of the devices in an Automatic Storage Management diskgroup should be the same size and have the same performance characteristics
- A diskgroup should not contain more than one partition on a single physical disk device.
- Using logical volumes as a device in an Automatic Storage Management diskgroup is not supported with Oracle RAC
- The user account with which you perform the installation (typically, 'oracle') must have write permissions to create the files in the path that you specify.

12. Shared Storage
Block    Device ASMlib Name Size Comments
DATA01 200G
SYSTEMDG 100G
RECO 200G


13. Partition the Shared Disks
Once the LUNs have been presented from the SAN to ALL servers in the cluster, "partition the LUNs from one node only", 
run fdisk to create a single whole-disk partition with exactly 1 MB offset on each LUN to be used as ASM Disk.
Tip: From the fdisk prompt, type "u" to switch the display unit from cylinder to sector. Then create a single
primary partition starting on sector 2048 (1MB offset assuming sectors of 512 bytes per unit). See below example for /dev/sda:

fdisk /dev/dm-2
Command (m for help): u
Changing display/entry units to sectors
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First sector (61-1048575, default 61): 
Last sector or +size or +sizeM or +sizeK (2048-1048575, default 1048575):
Using default value 1048575
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.

#partprobe

14.Installing and Configuring ASMLib
Download the following packages from the ASMLib OTN page.
NOTE: The ASMLib kernel driver MUST match the kernel revision number, the kernel revision number of
your system can be identified by running the "uname -r" command. Also, be sure to download the set of
RPMs which pertain to your platform architecture, in our case this is x86_64.

oracleasm-support-2.1.3-1.el5x86_64.rpm
oracleasmlib-2.0.4-1.el5.x86_64.rpm
oracleasm-2.6.18-92.1.17.0.2.el5-2.0.5-1.el5.x86_64.rpm

15. Configure ASMLib by running the following as the root user
a. NOTE: If using user and group separation for the installation (as documented here), the ASMLib driver
interface owner is 'grid' and the group to own the driver interface is 'asmadmin'. These groups were created in
section 2.1. If a more simplistic installation using only the Oracle user is performed, the owner will be 'oracle'
and the group owner will be 'dba'.

#/etc/init.d/oracleasm configure
Default user to own the driver interface []: oracle
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]

b.# Using ASMLib to Mark the Shared Disks as Candidate Disks [*** from single node]

#/usr/sbin/oracleasm createdisk disk_name device_partition_name

In this command, disk_name is the name you choose for the ASM disk. The name you choose must contain
only ASCII capital letters, numbers, or underscores, and the disk name must start with a letter, for example,
DISK1 or VOL1, or RAC_FILE1. The name of the disk partition to mark as an ASM disk is the
device_partition_name. For example:
# /usr/sbin/oracleasm createdisk DATA01_01 /dev/dm-2 #after linux partition in the above [step 13], dev/dm-2 became /dev/dm-2p1, but hare we need to use dev/dm-2
# /usr/sbin/oracleasm createdisk DATA01_02 /dev/dm-3
# /usr/sbin/oracleasm createdisk SYSTEMDG_01 /dev/dm-4
# /usr/sbin/oracleasm createdisk RECO_01  /dev/dm-5
# /usr/sbin/oracleasm createdisk RECO_02 /dev/dm-6

**Repeat step 1 for each disk that will be used by Oracle ASM

If you need to unmark a disk that was used in a createdisk command, you can use the following syntax as the root user:
# service oracleasm stop
# /usr/sbin/oracleasm deletedisk disk_name

c. *** On all the other nodes in the cluster, use the scandisks command as the root user to pickup the newly created ASM disks. 
You do not need to create the ASM disks on each node, only on one node in the cluster.
# /usr/sbin/oracleasm scandisks [all nodes]
Scanning system for ASM disks [ OK ]

d. After scanning for ASM disks, display the available ASM disks on each node to verify their availability:
# /usr/sbin/oracleasm listdisks
DATA01_01
DATA01_02
SYSTEMDG_01
RECO_01
RECO_01

[root@PBORADB1 ~]# /usr/sbin/oracleasm querydisk RECO_02
Disk "RECO_02" is a valid ASM disk
[root@PBORADB1 ~]# /usr/sbin/oracleasm querydisk /dev/dm-6
Device "/dev/dm-6" is marked an ASM disk with the label "RECO_02"
[root@PBORADB1 ~]#



16. Oracle Grid Infrastructure Install

a. unzip the gridinfra software zip file [on both node] [as grid owner]
    cd grip/rpm
    rpm -Uvh cvuqdisk*

b. As the grid user (Grid Infrastructure software owner) start the installer by running "runInstaller" from the staged installation media
c. Choose Skip Software Updates
d. Select radio button 'Install and Configure Grid Infrastructure for a Cluster' and click ' Next> '
e. Select radio button 'Advanced Installation' and click ' Next> 
f. Accept 'English' as language' and click  Next> 
g. Specify your cluster name and the SCAN name you want to use and click  Next> 
h. Uncheck Configure GNS
i. Use the Edit and Add buttons to specify the node names and virtual IP addresses you configured previously in your /etc/hosts file. 
   Use the 'SSH Connectivity' button to configure/test the passwordless SSH connectivity between your nodes.
j. Type in the OS password for the user 'oracle' and press 'Setup' and test and ok 'Next>'
k. Click on 'Interface Type' next to the Interfaces you want to use for your cluster and select the correct values for 'Public', 'Private' and 'Do Not Use' 
   When finished click ' Next> ' [we can use multiple NIC as private or public]
l. Select radio button Automatic Storage Management (ASM) and click ' Next> '
m. Select the 'DiskGroup Name' specify the 'Redundancy' and tick the disks you want to use, when done click ' Next> '
  [if no candidate disks found, then change discovery path to /dev/oracleasm/disks* or ORCL:*]
  NOTE: The number of voting disks that will be created depend on the redundancy level you specify:
  EXTERNAL will create 1 voting disk, NORMAL will create 3 voting disks, HIGH will create 5 voting disks   
n. Specify and conform the password you want to use and click ' Next> '
o. Select NOT to use IPMI and click ' Next> '
p. Assign groups as below:
   Oracle ASM DBA: asmdba
   Oracle ASM Operator: asmoper
   Oracle ASM Administrator: asmadmin
q. Specify the locations for your ORACLE_BASE and for the Software location and click ' Next> '
q. Specify the locations for your Inventory directory and click ' Next>'
r. Check that status of all checks is Succeeded and click ' Next> '
s. Wait for the OUI to complete its tasks
t. At this point you may need to run oraInstRoot.sh on all cluster nodes (if this is the first installation of an Oracle product on this system)
   Run them on the first node alone then go for other nodes
   Then press ok
   NOTE: root.sh should be run on one node at a time.
   if root.sh fails, then resolve it and run again
u. Wait for the OUI to finish the cluster configuration.

check OCR,Voting & OLR :
ocrcheck
crsctl query css votedisk
grid_home/cdata/.olr 


17. RDBMS Software Install   

a. #su - oracle
change into the directory where you staged the RDBMS software
./runInstaller

b. Provide your e-mail address, tick the check box and provide your Oracle Support Password 
   if you want to receive Security Updates from Oracle Support and click ' Next> ' 
c. Select the option 'Install Database software only' and click ' Next> '
d. Select 'Real Application Clusters database installation', and select all nodes. 
   Use the 'SSH Connectivity' button to configure/test the passwordless SSH connectivity between your nodes 
e. Type in the OS password for the oracle user and click 'Setup'   
f. To confirm English as selected language click ' Next> '
g. Make sure radio button 'Enterprise Edition'/'Standard Edition' is ticked, click ' Next> '
h. Specify path to your Oracle Base and below to the location where you want to store the software (Oraclehome). Click ' Next> '
i. Specify groups:
   Database Administrator: dba
   Database Operator:oinstall
j. Oracle Universal Installer performs prerequisite checks.
k. Check that the status of all checks is 'Succeeded' and click ' Next> '
l. Log in to a terminal window as root user and run the root.sh script on the first node. 
   When finished do the same for all other nodes in your cluster as well. When finished click 'OK'
   NOTE: root.sh should be run on one node at a time.
m. Click ' Close ' to finish the installation of the RDBMS Software.

18. Run ASMCA to create diskgroups

a. #su - grid
   cd /u01/11.2.0/grid/bin
   ./asmca
b. Click 'Create' to create a new diskgroup
c. Type in a name for the diskgroup, select the redundancy you want to provide and mark the tick box for the disks you want to assign to the new diskgroup   
d. Click 'OK'
e. Click 'Create' to create the diskgroup for the flash recovery area
f. Type in a name for the diskgroup, select the redundancy you want to provide and mark the tick box for the disks you want to assign to the new diskgroup.
g. Click 'OK'
h. Click 'Exit'

19. It is Oracles Best Practice to have an OCR mirror stored in a second diskgroup. 
    To follow this recommendation add an OCR mirror. Mind that you can only have one OCR in a diskgroup.
Action:
a. To add OCR mirror to an Oracle ASM diskgroup, ensure that the Oracle Clusterware stack is running and
run the following command as root from the GridInfrastructureHome? /bin directory:
b. # ocrconfig -add +ORADATA
c. # ocrcheck

20. Run DBCA to create the database
a. #su - oracle
   cd /u01/app/oracle/product/11.2.0/db_1/bin
   ./dbca
b. Select 'Oracle Real Application Clusters database' and click 'Next'
c. choose option 'Create a Database' and click 'Next'
d. Select the database template that you want to use for your database and click 'Next'
e. Type in the name you want to use for your database and select all nodes before you click 'Next'
f. select the options you want to use to manage your database and click 'Next'
g. Type in the passwords you want to use and click 'Next'
h. Select the diskgroup you created for the database files and click 'Multiplex Redo Logs and Control Files'. 
   In the popup window define the diskgroup that should contain controlfiles and 
   redo logfiles and the diskgroup that should contain the mirrored files.   
   When all file destinations are correct click 'Next'
i. Specify the diskgroup that was created for the flash recovery area and define the size. 
   If the size is smaller than recommended a warning will popup.   
j. Select if you want to have sample schemas created in your database and click 'Next'
k. Review and change the settings for memory allocation, characterset etc. according to your needs and click 'Next'
l. Review the database storage settings and click 'Next'
m. Make sure the tickbox 'Create Database' is ticked and click 'Finish'
n. Review the database configuration details again and click 'OK'
o. The database is now created, you can either change or unlock your passwords or just click Exit to finish the database creation


Useful references:

http://www.oracleangels.com/2011/05/grid-infrasturuer-redundant.html
https://forums.oracle.com/forums/thread.jspa?threadID=2126077
http://logicalshift.blogspot.com/2010/05/linux-udev-and-multipath.html
MOS Doc:INS-20802 PRVF-9802 PRVF-5184 PRVF-5186 After Successful Upgrade to 11gR2 Grid Infrastructure [ID 974481.1]
MOS Doc: RAC and Oracle Clusterware Best Practices and Starter Kit (Platform Independent) [ID 810394.1]
https://forums.oracle.com/forums/thread.jspa?threadID=2156504
http://www.oracle.com/technetwork/topics/linux/asmlib/index-101839.html

Restore/Clone a Database from HP DP

I also have a similar post, which was using RMAN command from the target and using autonabcup controlfile. But this one is using "CONTROL FILE FROM DP MANAGED BACKUP"
We were planning to restore our production database to our test machine as a preparation of DR. Our current environment was as below:
- Source/Production Datbase on HPUX and Oracle 9.2
- HP Data Protector version 5.5
- RMAN Backup was taken with pool 1. FULL & INCREMENTAL BACKUP 2. Archivelog Daily Backup

1. Prepare the partitons of target machine similar to the Source/Production machine.
2. Make sure that all the needed media are on the tape library [or available an if mount request prompts you can change with the required tape]
3. If Partitions are not similar then create soft links considering space requirements.
4. Install Oracle Database SW [Same as production]
5. Configure DP client on the target machine
7. Prepare a pfile [initMYDB.ora] mentioning the locations of control files and archivelog files and other info line PGA, SGA size etc.
8. Execute below commands :

$ export ORACLE_SID=MYDB
$ rman target /
RMAN > startup nomount

9. Create listener serving the new database [MYDB]
10. Create a TNS enty for the new database [MYDB]
11. Create Password file for the the target database [MYDB]
12. Check that you can connect using password file:

$ sqlplus /nolog
SQL> conn sys/[password as in passwordfile]@MYDB as sysdba

13. From DP follow options to restore[to file system] controlfile from DP managed backup of control file:
                          > restore
                          > Oracle Server
                          > [source/production machine]
                          > click on [source/production DATABSE, in mycase it is MYDB]
                          > Select "Source" from the Tab Pane appeared on the right side
                          > From restore sction menu select "Perform RMAN Repository Restore"
                          > Select radio button "CONTROL FILE FROM DP MANAGED BACKUP"
                          > Select "Option" Tab
                          > From "Restore to client" menu select the target Machine
                          > Click the "Settings" Button and give username:sys password & service [in my case MYDB] then OK
                          > Cilck the "Device" tab and select your drive
                          > Enter the OS user & group whivh owns the oracle SW
                          > From the "Session ID" menu select the latest possible session id[or select as your need]
                          > CLick "Restore" button

Now, DP will restore the control file in the tmp directory of omni like below:                         
/var/opt/omni/tmp/ctrl_MYDB.dbf

14. Now restore the controlfile for the target DB as below:
$ export ORACLE_SID=MYDB
RMAN > run {
allocate channel 'dev0' type disk;
restore controlfile from '/var/opt/omni/tmp/ctrl_MYDB.dbf ';
release channel 'dev0';
}
Now, control file are restored to the locations specified in pfile of MYDB

15. now mount the database [as now we got control files]:

$ export ORACLE_SID=MYDB
RMAN > alter database mount;

16. From DP follow options to restore database from the full backup:
                          > restore
                          > Oracle Server
                          > [source/production machine]
                          > click on [source/production DATABSE, in mycase it is MYDB]
                          > Select "Source" from the Tab Pane appeared on the right side
                          > From restore sction menu select "Perform Restore"
                          > Check "DATABASE" option from the root [if you want to restore the full database]
                          > Select "Option" Tab
                          > From "Restore to client" menu select the target Machine
                          > Enter the OS user & group which owns the oracle SW                         
                          > Click the "Settings" Button and give username:sys password & service [in my case MYDB] then OK
                          > Let the "Restore Mode" to normal
                          > Set parallelism 4
                          > Slecet "Restore Until" option according to your need
                          > Cilck the "Device" tab and select your drive
                          > CLick "Restore" button

Now, DP will detech tapes from the library[or mount request will prompt] and restore all the datafiles to the location accrofing to the control file.

17. From DP follow options to recover database:
                          > restore
                          > Oracle Server
                          > [source/production machine]
                          > click on [source/production DATABSE, in mycase it is MYDB]
                          > Select "Source" from the Tab Pane appeared on the right side
                          > From restore sction menu select "Perform Recovery Only"
                          >  "DATABASE" option Check autometically [no need to do anything]
                          > Select "Option" Tab
                          > From "Restore to client" menu select the target Machine
                          > Enter the OS user & group which owns the oracle SW                         
                          > Click the "Settings" Button and give username:sys password & service [in my case MYDB] then OK
                          > Let the "Restore Mode" to normal
                          > Set parallelism 4
                          > Slecet "Recover Until" option according to your need
                          > Cilck the "Device" tab and select your drive
                          > I put "Open database after recover" unchecked
                          > CLick "Restore" button
                         
- DP will recever with incrimental backups first and then restore archivelogs to the location on archivelof in initMYDB.ora location and apply them
- if you have space constraints then move the related archive logs to another location after message like below on DP restore message:
archive log filename=/billing/mydb/arch/MYDB/MYDB_56595.arc thread=1 sequence=56595
archive log filename=/billing/mydb/arch/MYDB/MYDB_56596.arc thread=1 sequence=56596
archive log filename=/billing/mydb/arch/MYDB/MYDB_56597.arc thread=1 sequence=56597

18. Now open the database:

$ export ORACLE_SID=MYDB
RMAN> alter database open resetlogs;                       

- Now, database should be fully recovered and consistant and open
- Check alert log for error and resolve them

19. Create and adjust temp tablespace as per your need.

20. Now optionally do below:
     - change the database name
     - create a password file for the new name
     - Correct the TNS and listener for the new name
     - startup the databse
     - create spfile and reboot the DB
     - check the alertlog carefully


Some of the problems that may occur during above process are given below:

i. validation failed for archived log:

In alert log we may see below error:
recover database until time "to_date('10/26/11 00:00:00','MM/DD/YY HH24:MI:SS')";

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 10/27/2011 21:23:25
RMAN-06053: unable to perform media recovery because of missing log
RMAN-06025: no backup of log thread 1 seq 56617 scn 3098608773567 found to restore
...............
...............
RMAN-06025: no backup of log thread 1 seq 56605 scn 3098608377404 found to restore

Cause: Recover,while applying archivelog, failed due to space limitation or other cause

Solution:
1. Again try to recover database from DP after the resolving the associate problem.
         
2. manually copy archivelogs from the production or from other source and do as below:

2.a crosscheck archivelogs:
RMAN> crosscheck archivelog all;

while crosschecking if you error like below:
.......
.......
validation failed for archived log
archive log filename=/billing/mydb/arch/log_archive/MYDB_56618.arc recid=56167 stamp=765500466

2.b then, first check the current archivelog location of the target database and make sure that archivelogs names associated with the error exisrs there.
this occurs becaus the location of the archivelog of source and terget is not same [may occur when archivelogs are copied manually]
To resolve above we need no catalog archivelogs as below for all the archivelogs archivelogs gave error:

RMAN> catalog archivelog '/billing/mydb/arch/MYDB/MYDB_56618.arc';
...
...
RMAN> catalog archivelog '/billing/mydb/arch/MYDB/MYDB_56618.arc';

2.c then crosscheck again:
RMAN> crosscheck archivelog all;

Now, we if we get both success and fail for the same archivelog as below then it is ok, just ignore the error message:
validation succeeded for archived log
archive log filename=/billing/mydb/arch/MYDB/MYDB_56618.arc recid=56230 stamp=765667425
validation failed for archived log
archive log filename=/billing/mydb/arch/log_archive/MYDB_56618.arc recid=56167 stamp=765500466

2.d Now try to recover again:

RMAN> recover database until time "to_date('10/26/11 00:00:00','MM/DD/YY HH24:MI:SS')";

Starting recover at 27-OCT-11
using channel ORA_DISK_1

starting media recovery

archive log thread 1 sequence 56605 is already on disk as file /billing/mydb/arch/MYDB/MYDB_56605.arc
...
...
archive log thread 1 sequence 56618 is already on disk as file /billing/mydb/arch/MYDB/MYDB_56618.arc

archive log filename=/billing/mydb/arch/MYDB/MYDB_56605.arc thread=1 sequence=56605
....
....
archive log filename=/billing/mydb/arch/MYDB/MYDB_56619.arc thread=1 sequence=56619
media recovery complete
Finished recover at 27-OCT-11


2.e ignore nelow message on alertlog:

ORA-279 signalled during: alter database recover logfile '/billing/mydb/...'

2.f now to eliminate "both success and fail for the same archivelog" in "crosscheck archivelog all" execute below:

RMAN> crosscheck archivelog all;
RMAN> delete expired archivelog all;

ii. At the time of "alter database open resetlogs", observe below:

RMAN> alter database open resetlogs;

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of alter db command at 10/27/2011 21:32:57
ORA-01092: ORACLE instance terminated. Disconnection forced

1. ingnore below alertlog:
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00312: online log 2 thread 1: '/billing/mydb/redo1/oradata/redo_02a.log'
ORA-27037: unable to obtain file status

2. but "ORA-01092: ORACLE instance terminated. Disconnection forced" is a issue, Please check the alert log:

ORA-30012: undo tablespace 'UNDOTBS1' does not exist or of wrong type

To resolve above error , startup mount do as below:

2.a
SQL> select name from v$tablespace;

NAME
------------------------------
SYSTEM
TBS_BILLIMAGES
TBS_ORDERHDR
.......
IDX_ORDERHDR
TEMP
TBS_SMALL
UNDOTBS  -- <<<<<<< So, we donot have tablespace 'UNDOTBS1', our undo tablesace name is 'UNDOTBS'

39 rows selected.

2.b Now, correct this as below:

SQL> show parameter undo;

NAME                                 TYPE                             VALUE
------------------------------------ -------------------------------- ------------------------------
undo_management                      string                           AUTO
undo_retention                       integer                          21600
undo_suppress_errors                 boolean                          FALSE
undo_tablespace                      string                           UNDOTBS1
SQL>

SQL> shutdown immediate;

2.c [ optionally create a new one and ] edit initMYDB.ora and set "undo_tablespace=UNDOTBS" and startup using the pfile:

SQL> startup pfile='location of the pfile';
SQL> create spfile from pfile='location of the pfile';

thanks to below links:
http://repettas.wordpress.com/2008/11/01/restore-archive-logs-to-new-location-using-rman/
http://www.dbasupport.com/forums/showthread.php?t=53108
https://forums.oracle.com/forums/thread.jspa?threadID=636113