This was my RAC installation on around september last year on HPUX.
Here I am sharing my action plan as it is I prepared on notepad.
Oracle Version : 11.2.0.3
OS : HPUX Itanium 11.31
1. configure NTP
2. configure DNS using SAM
3 . Create OS groups using the command below. Enter these commands as the 'root' user:
/usr/sbin/groupadd -g 503 oinstall
/usr/sbin/groupadd -g 504 dba
/usr/sbin/groupadd -g 505 asmadmin
/usr/sbin/groupadd -g 506 asmdba
/usr/sbin/groupadd -g 507 asmoper
4. Create the users that will own the Oracle software using the commands:
*** please use "/usr/bin/ksh" shell for oracle user
#/usr/sbin/useradd -u 503 -s /usr/bin/ksh -g oinstall -G dba,asmadmin,asmdba,asmoper -p /home/oracle oracle
5. Set the password for the oracle account. [Oracle6!7]
6. Sample /etc/hosts file will be as below: "**also configure DNS with reverse resolve":
127.0.0.1 localhost.localdomain localhost #dont add anyhing extra in this line
#lan1 - PUBLIC
10.101.4.57 ORAdb1.robi.com.bd ORAdb1
10.101.4.58 ORAdb2.robi.com.bd ORAdb2
#VIP
10.101.4.59 ORAdb1-vip.robi.com.bd ORAdb1-vip
10.101.4.60 ORAdb2-vip.robi.com.bd ORAdb2-vip
#lan3 - PRIVATE
172.16.4.57 ORAdb1-pvt
172.16.4.58 ORAdb2-pvt
7. Kernel parameters: (can also be fixed during installation using runfixup.sh)
ksi_alloc_max 32768
executable_stack 0
ksi_alloc_max 32768
max_thread_proc 1024
maxdsiz 1073741824 (1 GB)
maxdsiz_64bit 2147483648 (2 GB)
maxfiles 16384
maxfiles_lim 65536
maxssiz 134217728 (128 MB)
maxssiz_64bit 1073741824 (1 GB)
maxuprc 3686
msgmni 4096
msgtql 4096
ncsize 35840
nflocks 4096
ninode 34816
nkthread 7184
nproc 4096
semmni 4096
semmns 8192
semmnu 4096
semvmx 32767
shmmax 1073741824
shmmni 4096
shmseg 512
tcp_smallest_anon_port 9000
tcp_largest_anon_port 65500
udp_smallest_anon_port 9000
udp_largest_anon_port 65500
8. install below packages or upper versions:
Operating system:
HP-UX 11iV3 patch Bundle Sep/ 2008 (B.11.31.0809.326a)
HP-UX 11.31 packages and bundles:
PHCO_41479 (or later) 11.31 diskowner(1M) cumulative patch
PHKL_38038 vm cumulative patch
PHKL_38938 11.31 SCSI cumulative I/O patch
PHKL_40941 Scheduler patch : post wait hang
PHSS_36354 11.31 assembler patch
PHSS_37042 11.31 hppac (packed decimal)
PHSS_37959 Libcl patch for alternate stack issue fix (QXCR1000818011)
PHSS_39094 11.31 linker + fdp cumulative patch
PHSS_39100 11.31 Math Library Cumulative Patch
PHSS_39102 11.31 Integrity Unwind Library
PHSS_38141 11.31 aC++ Runtime
To avoid the node evictions:
PHKL_40208
PHKL_40372
HP-UX Logical Volume Manager (LVM):
PHCO_41479 (or later) 11.31
C / C++ compiler:
HP C/aC++ A.06.20 (Swlist Bundle - C.11.31.04) - September 2008
C Compiler Patches:
PHSS_39824 11.31 HP C/aC++ Compiler (A.06.23)
PHSS_39826 11.31 u2comp/be/plug-in (C.06.23)
Gcc Compiler:
Gcc 4.2.3
X11MotifDevKit.MOTIF21-PRG
Oracle JDBC/OCI Drivers:
HPUX JDK 6.0.05
HPUX JDK 5.0.15
Oracle ODBC:
You can download and install the Driver Manager from the following URL:http://www.unixodbc.org
IBM MQ Series V. 6.0, client and server (optional):
MQSERIES.MQM-CL-HPUX
MQSERIES.MQM-SERVER
Motif 2.1 Development Environment package (optional):
TIBCO Rendezvous 7.2
Perl:
Perl 5.8.8
Programming Software (optional):
Pro*Cobol - Micro Focus Server Express 5.1
Pro*FORTRAN - HP FORTRAN/90 - Sep 2008 release
SSH:
The required SSH software is the default SSH shipped with your operating system.
10 Creating Required Symbolic Links
This task is required only if the Motif 2.1 Development Environment package (X11MotifDevKit.MOTIF21-PRG) is not installed.
# cd /usr/lib
ln -s libX11.3 libX11.sl
ln -s libXIE.2 libXIE.sl
ln -s libXext.3 libXext.sl
ln -s libXhp11.3 libXhp11.sl
ln -s libXi.3 libXi.sl
ln -s libXm.4 libXm.sl
ln -s libXp.2 libXp.sl
ln -s libXt.3 libXt.sl
ln -s libXtst.2 libXtst.sl
11. Create the Oracle Inventory Directory
mkdir -p /u01/app/oraInventory
chown -R oracle:oinstall /u01/app/oraInventory
chmod -R 775 /u01/app/oraInventory
12. Creating the Oracle Grid Infrastructure Home Directory
mkdir -p /u01/app/11.2.0/grid
chown -R oracle:oinstall /u01/app/11.2.0/grid
chmod -R 775 /u01/app/11.2.0/grid
vi oracrs.sh
export ORACLE_SID=+ASM1/+ASM2
export ORACLE_HOME=/u01/app/11.2.0/grid
export PATH=$ORACLE_HOME/bin:$PATH
13. Creating the Oracle Base Directory
mkdir -p /u01/app/oracle/product/11.2.0/dbhome_1
chown -R oracle:oinstall /u01/app/oracle/
chmod -R 775 /u01/app
vi oradb.sh
export ORACLE_SID=ORADB1/ORADB2
export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
export PATH=$ORACLE_HOME/bin:$PATH
su - oracle
vi .profile and append below on both node:
export ORACLE_HOSTNAME=ORAdb1.robi.com.bd
14. ASM DIsk Group sizing
DATA01 200G
SYSTEMDG 100G
RECO 100G
15. Configuring Disk Devices for Oracle ASM
a. To make sure that the disks are available, enter the following command on every node:
# ioscan -fNnC disk
b. If the ioscan command does not display device name information for a device, enter the following command to install the special device files for any new devices:
# /usr/sbin/insf -e
c. For each disk to add to a disk group, enter the following command on any node to verify that it is not already part of an LVM volume group:
# /sbin/pvdisplay /dev/dsk/cxtydz
If this command displays volume group information, the disk is already part of a volume group.
The disks that you choose must not be part of an LVM volume group.
d. The disk names for the same disk can be different on each node. A disk can be identified as the same
one via the WWID. The WWID of a disk can be checked via the following command:
# scsimgr lun_map -D /dev/rdisk/disk29 | grep WWID
ORADB1:/home/root #scsimgr lun_map -D /dev/rdisk/disk29 | grep WWID
World Wide Identifier(WWID) = 0x6006016001102d00b81efc7444e1e111
ORADB2:/home/root #scsimgr lun_map -D /dev/rdisk/disk30 | grep WWID
World Wide Identifier(WWID) = 0x6006016001102d00b81efc7444e1e111
ORADB1:/home/root #scsimgr lun_map -D /dev/rdisk/disk30 | grep WWID
World Wide Identifier(WWID) = 0x6006016001102d00b21efc7444e1e111
ORADB2:/home/root #scsimgr lun_map -D /dev/rdisk/disk29 | grep WWID
World Wide Identifier(WWID) = 0x6006016001102d00b21efc7444e1e111
ORADB1:/home/root #scsimgr lun_map -D /dev/rdisk/disk31 | grep WWID
World Wide Identifier(WWID) = 0x6006016001102d00b61efc7444e1e111
ORADB2:/home/root #scsimgr lun_map -D /dev/rdisk/disk31 | grep WWID
World Wide Identifier(WWID) = 0x6006016001102d00b61efc7444e1e111
ORADB1:/home/root #scsimgr lun_map -D /dev/rdisk/disk32 | grep WWID
World Wide Identifier(WWID) = 0x6006016001102d00ba1efc7444e1e111
ORADB2:/home/root #scsimgr lun_map -D /dev/rdisk/disk32 | grep WWID
World Wide Identifier(WWID) = 0x6006016001102d00ba1efc7444e1e111
e. create a special Oracle device directory and to use mknod to create device paths in this special Oracle folder
# mkdir /dev/oracle
# ll /dev/rdisk/disk29
crw-r----- 1 bin sys 23 0x000019 Jan 16 12:16 /dev/dsk/cxtydz
# mknod /dev/oracle/asmdisk1 c 23 0x000019
on ORAdb1
---------
#ll /dev/rdisk/disk29
#mknod /dev/oracle/asm_systemdg_01 c 13 0x000013
#ll /dev/rdisk/disk30
#mknod /dev/oracle/asm_data01_01 c 13 0x000014
#ll /dev/rdisk/disk31
#mknod /dev/oracle/asm_data01_02 c 13 0x000015
#ll /dev/rdisk/disk32
#mknod /dev/oracle/asm_reco_01 c 13 0x000016
Later during set-up, for the ASM instance, set the ASM_DISKSTRING parameter to /dev/oracle/* #*/
f. Modify the owner, group, and permissions on the character raw device files (ASM and Voting Disks) on all nodes:
# chown -R oracle:asmadmin /dev/oracle #*/
# chmod -R 660 /dev/oracle #*/
16. Setting the Minor Number for Device Files
a. Log in as the root user.
b.Determine whether /dev/async exists. If the device does not exist, then use the following command to create it:
# /sbin/mknod /dev/async c 101 0x4
c. Alternatively, you can set the minor number value to 0x104 using the following command:
# /sbin/mknod /dev/async c 101 0x104
d. If /dev/async exists, then determine the current value of the minor number, as shown in the following example:
#ll -l /dev/async
crw-rw-rw- 1 bin bin 101 0x000000 Jul 26 15:45 /dev/async
e. If the existing minor number of the file is not 0x4 or 0x104, then change it to an expected value using one of the following commands:
# rm /dev/async
# mknod /dev/async c 101 0x4
or
# mknod /dev/async c 101 0x104
17. Oracle Grid Infrastructure Install
a. As the grid user (Grid Infrastructure software owner) start the installer by running "runInstaller" from the staged installation media
b. Choose Skip Software Updates
c. Select radio button 'Install and Configure Grid Infrastructure for a Cluster' and click ' Next> '
d. Select radio button 'Advanced Installation' and click ' Next>
e. Accept 'English' as language' and click Next>
f. Specify your cluster name and the SCAN name you want to use and click Next>
g. Uncheck Configure GNS
h. Use the Edit and Add buttons to specify the node names and virtual IP addresses you configured previously in your /etc/hosts file.
Use the 'SSH Connectivity' button to configure/test the passwordless SSH connectivity between your nodes.
*** some times need to configure ssh manualy
i. Type in the OS password for the user 'oracle' and press 'Setup' and test and ok 'Next>'
j. Click on 'Interface Type' next to the Interfaces you want to use for your cluster and select the correct values for 'Public', 'Private' and 'Do Not Use'
When finished click ' Next> ' [we can use multiple NIC as private or public]
k. Select radio button Automatic Storage Management (ASM) and click ' Next> '
l. Select the 'DiskGroup Name' specify the 'Redundancy' and tick the disks you want to use,[if candidate disks are not shown then check permission for the /dev... folders] when done click ' Next> '
[if no candidate disks found, then change discovery path to /dev/oracleasm/disks* or ORCL:*]
NOTE: The number of voting disks that will be created depend on the redundancy level you specify:
EXTERNAL will create 1 voting disk, NORMAL will create 3 voting disks, HIGH will create 5 voting disks
*** if disk are ok on both node and it is sure that "/dev/oracle/.." names are pointing to the same disk device hen ignore "PRVF-5150" error
m. Specify and conform the password you want to use and click ' Next> '
n. Select NOT to use IPMI and click ' Next> '
o. Assign groups as below:
Oracle ASM DBA: asmdba
Oracle ASM Operator: asmoper
Oracle ASM Administrator: asmadmin
p. Specify the locations for your ORACLE_BASE and for the Software location and click ' Next> '
q. Specify the locations for your Inventory directory and click ' Next>'
r. Check that status of all checks is Succeeded and click ' Next> '
s. Wait for the OUI to complete its tasks
t. At this point you may need to run oraInstRoot.sh on all cluster nodes (if this is the first installation of an Oracle product on this system)
Run them on the first node alone then go for other nodes
Then press ok
NOTE: root.sh should be run on one node at a time.
if root.sh fails, then resolve it and run again
u. Wait for the OUI to finish the cluster configuration.
check OCR,Voting & OLR :
ocrcheck
crsctl query css votedisk
grid_home/cdata/
18. RDBMS Software Install
a. #su - oracle
change into the directory where you staged the RDBMS software
./runInstaller
b. Provide your e-mail address, tick the check box and provide your Oracle Support Password
if you want to receive Security Updates from Oracle Support and click ' Next> '
c. Select the option 'Install Database software only' and click ' Next> '
d. Select 'Real Application Clusters database installation', and select all nodes.
Use the 'SSH Connectivity' button to configure/test the passwordless SSH connectivity between your nodes
e. Type in the OS password for the oracle user and click 'Setup'
f. To confirm English as selected language click ' Next> '
g. Make sure radio button 'Enterprise Edition'/'Standard Edition' is ticked, click ' Next> '
h. Specify path to your Oracle Base and below to the location where you want to store the software (Oraclehome). Click ' Next> '
i. Specify groups:
Database Administrator: dba
Database Operator:oinstall
j. Oracle Universal Installer performs prerequisite checks.
k. Check that the status of all checks is 'Succeeded' and click ' Next> '
l. Log in to a terminal window as root user and run the root.sh script on the first node.
When finished do the same for all other nodes in your cluster as well. When finished click 'OK'
NOTE: root.sh should be run on one node at a time.
m. Click ' Close ' to finish the installation of the RDBMS Software.
19. Run ASMCA to create diskgroups
a. #su - grid
cd /u01/11.2.0/grid/bin
./asmca
b. Click 'Create' to create a new diskgroup
c. Type in a name for the diskgroup, select the redundancy you want to provide and mark the tick box for the disks you want to assign to the new diskgroup
d. Click 'OK'
e. Click 'Create' to create the diskgroup for the flash recovery area
f. Type in a name for the diskgroup, select the redundancy you want to provide and mark the tick box for the disks you want to assign to the new diskgroup.
g. Click 'OK'
h. Click 'Exit'
20. It is Oracles Best Practice to have an OCR mirror stored in a second diskgroup.
To follow this recommendation add an OCR mirror. Mind that you can only have one OCR in a diskgroup.
Action:
a. To add OCR mirror to an Oracle ASM diskgroup, ensure that the Oracle Clusterware stack is running and
run the following command as root from the GridInfrastructureHome? /bin directory:
b. # ocrconfig -add +DATA1
c. # ocrcheck
21. Run DBCA to create the database
a. #su - oracle
cd /u01/app/oracle/product/11.2.0/db_1/bin
./dbca
b. Select 'Oracle Real Application Clusters database' and click 'Next'
c. choose option 'Create a Database' and click 'Next'
d. Select the database template that you want to use for your database and click 'Next'
e. Type in the name you want to use for your database and select all nodes before you click 'Next'
f. select the options you want to use to manage your database and click 'Next'
g. Type in the passwords you want to use and click 'Next'
h. Select the diskgroup you created for the database files and click 'Multiplex Redo Logs and Control Files'.
In the popup window define the diskgroup that should contain controlfiles and
redo logfiles and the diskgroup that should contain the mirrored files.
When all file destinations are correct click 'Next'
i. Specify the diskgroup that was created for the flash recovery area and define the size.
If the size is smaller than recommended a warning will popup.
j. Select if you want to have sample schemas created in your database and click 'Next'
k. Review and change the settings for memory allocation, characterset etc. according to your needs and click 'Next'
l. Review the database storage settings and click 'Next'
m. Make sure the tickbox 'Create Database' is ticked and click 'Finish'
n. Review the database configuration details again and click 'OK'
o. The database is now created, you can either change or unlock your passwords or just click Exit to finish the database creation
22. For all the luns added in ASM, set LV timeout value = (# of paths * PV Timeout) + 10 seconds, using "lvchange -t timeinsec /dev/dsk/cxtydz" command
in our case, EMC recommended PV timeout = 90
# of path = 4
then check with # lvdisplay -v /dev/dsk/cxtydz
Sample change cluster resource
------------------------
ex: change listener port
------------------------
emca -deconfig dbcontrol db -repos drop -cluster
- srvctl modify listener -l
- restart the full cluster [ crsctl stop/start crs]
- check with "lsnrctl status", from both DB & AMS "show parameter local"
emca -config dbcontrol db -repos create -cluster
-----------------------------
ex: change scan listener port
-----------------------------
emca -deconfig dbcontrol db -repos drop -cluster
- srvctl modify scan_listener -p 1521
- alter system set remote_listener='ORAdb-scan.robi.com.bd:1521' scope=both;--from DB & ASM on all the db nodes
- Change the necessary files in $ORACLE_HOME/network/admin on server & client
- restart the full cluster [ crsctl stop/start crs]
- check with "lsnrctl status", from both DB & AMS "show parameter remote"
- from all DB nodes "lsnrctl status LISTENER", also check same for all scan lsitener
emca -config dbcontrol db -repos create -cluster
---------------
emca -deconfig dbcontrol db -repos drop -cluster
emca -config dbcontrol db -repos create -cluster
http://docs.oracle.com/cd/E11882_01/install.112/e10815/typinstl.htm#BABFEGHH
http://docs.oracle.com/cd/E11882_01/install.112/e10815/pre_hpux.htm#CIHFICFD
http://docs.oracle.com/cd/E11882_01/install.112/e10815/manpreins.htm#BABCHAED
http://docs.oracle.com/cd/E11882_01/install.112/e10815/pre_hpux.htm#CEGGJDIG
http://docs.oracle.com/cd/E11882_01/install.112/e10815/storage.htm#CDEECFGI
http://gjilevski.wordpress.com/2010/10/03/fresh-oracle-11-2-0-2-grid-infrastructure-installation-prvf-5150-prvf-5184/
MOS Doc: RAC and Oracle Clusterware Best Practices and Starter Kit (Platform Independent) [ID 810394.1]
Thannk you very much!
ReplyDelete