Disaster Recovery using HP Data Protector and Oracle RMAN

In our case we have used below components:
- HP Data Protector Version 6.2
- Source & Destination System: RHEL5.5
- Source & Destination Database: Oracle 11.2.0.1

Scenario:
- Our production [source] machine is completely out of order and inaccessible
- We have scheduled RMAN Backup [incremental] on TAPE which is configured with HPDP 6.2
- RMAN Backups are taken with "CONTROLFILE AUTOBACKUP ON;" configuration of RMAN
[if "CONTROLFILE AUTOBACKUP OFF;" then we need to create a pfile manually]
- Our Database name is MYDB and Backup Specifications names for this DB starts with MYDB_*
- Machine of Cell Manager is also completely damaged
- We have backup of HPDP iDB on tape
- So, currently we only have Tapes containing HPDP iDB latest backup & RMAN backup


So we need to do following:
1. Configure a new Cell Manager
2. Scan load tape information for the Tape containing DP iDB backup
3. import iDB from the backup on tape
4. Scan Tapes containing RMAN backup
5. Configure a new machine with same OS and Connectivity with new Cell Manager
6. Create the partitions locations as much as similar to the old machine as possible
7. Install Oracle SW only of same version [version means consideration 32bit and 64bit]
8. Create a LISTENER listening on default port
9. Install and configure HP DP 6.2 Client on the target DB machine [Install HP Data Protector 6.2 Client [and use in RMAN]]
10. Restore and Recover the Oracle Database named MYDB on new machine

Now we are going to discuss step 10 in details:

[If you are on windows create a oracle DB service for MYDB as below on new machine before proceeding:
Please run the "Command Prompt" right click and "run as administrator" if current user is not "administrator"
oradim -new -sid MYDB -startmode m
]

For all the below steps, make sure that desired ORACLE_HOME is exported correctly

a. Collection configuration information:

- identify the backup spec that contains latest backup in our case [NODE1_MYDB_INCR1]
- from the latest rman backup session of DP [internal database>sessions>properties>message] collect below to parameter:
dbid 4261461017;
parms 'SBT_LIBRARY=/opt/omni/lib/libob2oracle8_64bit.so,ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=MYDB,OB2BARLIST=NODE1_MYDB_INCR1)';

- identify and crosscheck needed media from session messages using lines like below:
Initializing new medium: "Default File_6"
Loading medium from slot [1709a8c054df607235202050011.fd] to device [Test_01_Writer0]

b. Restore spfile & controlfile backedup by "CONTROLFILE AUTOBACKUP" from Tape using below steps:

$ export ORACLE_SID=MYDB
$ rman target /
connected to target database (not started)
[ignore above, because we do not have any Database now]

RMAN> startup nomount

startup failed: ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/oracle1/orabase/product/11.2.0/dbhome_1/dbs/initMYDB.ora'
--ignore above, because we donot have s/pfile and DUMMY DB is needed now

RMAN>
set dbid 4261461017;
run {
allocate channel 'dev_0' type 'sbt_tape'
parms 'SBT_LIBRARY=/opt/omni/lib/libob2oracle8_64bit.so,ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=MYDB,OB2BARLIST=NODE1_MYDB_INCR1)';
restore spfile from autobackup;
restore controlfile from autobackup;
release channel 'dev_0';
}

allocated channel: dev_0
channel dev_0: SID=67 device type=SBT_TAPE
channel dev_0: Data Protector A.06.20/370

Starting restore at 12-JUN-11

channel dev_0: looking for AUTOBACKUP on day: 20110612
channel dev_0: AUTOBACKUP found: c-2169732094-20110612-01
channel dev_0: restoring spfile from AUTOBACKUP c-2169732094-20110612-01
channel dev_0: SPFILE restore from AUTOBACKUP complete
Finished restore at 12-JUN-11

Starting restore at 12-JUN-11

channel dev_0: looking for AUTOBACKUP on day: 20110612
channel dev_0: AUTOBACKUP found: c-2169732094-20110612-01
channel dev_0: restoring control file from AUTOBACKUP c-2169732094-20110612-01
channel dev_0: control file restore from AUTOBACKUP complete
output file name=/oracle1/orabase/product/11.2.0/dbhome_1/dbs/cntrlMYDB.dbf --<< control file restored location
Finished restore at 12-JUN-11

released channel: dev_0

[controlfile and spfile will be restored to the default location $ORACLE_HOME/dbs]

c. Then create a pfile by editing the restored spfile [change the location and other parameters]

d. shutdown the Database and start the database using the newly created pfile:

$ export ORACLE_SID=MYDB
$ sqlplus / as sysdba
SQL> shutdown immediate

SQL> startup nomount pfile='[location of the newly created pfile]'

e. Change the locations for controlfile or copy controlfiles to the desired location mentioned in the in pfile and mount the Database
[or we could create the pfile with the modified "control_files" parameter]

SQL> show parameter control_files

control_files string /system1/oradata/MYDB/cntl/c
ontrol01.ctl, /data01/oradata/
MYDB/cntl/control02.ctl

SQL>! cp $ORACLE_HOME/dbs/cntrlMYDB.dbf /system1/oradata/MYDB/cntl/control01.ctl
SQL>! cp $ORACLE_HOME/dbs/cntrlMYDB.dbf /data01/oradata/MYDB/cntl/control02.ctl

SQL> alter database mount


f. after mount we can restore it from DP or using rman as below,
"tape identification and loading unloading will be handled by DP, please monitor session on DP"

$ export ORACLE_SID=MYDB
$ target /

RMAN> run {
allocate channel 'dev_0' type 'sbt_tape'
parms 'SBT_LIBRARY=/opt/omni/lib/libob2oracle8_64bit.so';
restore database;
}
[Here only SBT_LIBRARY parms is needed, as rest of the info is now in restored control files.]

If you find error like below then try to create same locations or restore to new location:
ORA-19504: failed to create file "......."
ORA-27040: file create error, unable to create file
Linux-x86_64 Error: 2: No such file or directory

Use below command to identify missing datafiles and original locations:
RMAN> VALIDATE DATABASE;
RMAN> REPORT SCHEMA;

if same location is create then:
run {
allocate channel 'dev_0' type 'sbt_tape'
parms 'SBT_LIBRARY=/opt/omni/lib/libob2oracle8_64bit.so';
restore datafile 15;
release channel 'dev_0';
}

if new location is needed then:
run {
SET NEWNAME FOR DATAFILE 1 TO '[new location for datafile 1]';
SET NEWNAME FOR DATAFILE 5 TO '[new location for datafile 5]';
SET NEWNAME FOR DATAFILE 15 TO '[new location for datafile 15]';
allocate channel 'dev_0' type 'sbt_tape'
parms 'SBT_LIBRARY=/opt/omni/lib/libob2oracle8_64bit.so';
restore datafile 1;
restore datafile 5;
restore datafile 15;
SWITCH DATAFILE ALL;
release channel 'dev_0';
}

g. Collect the latest SCN [as redo logs are also lost] which will be user to recover.
This will eliminate the porbability of RMAN-06054 & RMAN-06556

$ export ORACLE_SID=MYDB
SQL> select group#, first_change#, status, archived from v$log order by FIRST_CHANGE# desc;

GROUP# FIRST_CHANGE# STATUS ARC
---------- ------------- ---------------- ---
2 968291 CURRENT NO ---<<< target SCN, SCN just after the latest archived redolog
1 968224 ACTIVE YES
3 963378 INACTIVE YES


i. Recover Database to the latest SCN

$ export ORACLE_SID=MYDB
$ rman target /
RMAN> run {
allocate channel 'dev_0' type 'sbt_tape'
parms 'SBT_LIBRARY=/opt/omni/lib/libob2oracle8_64bit.so';
recover database until scn 968291;
}
[Here only SBT_LIBRARY parms is needed, as rest of the info is now in restored control files.]


j. Open database using "resetlogs" option [as redo logs are lost]

RMAN> alter database open resetlogs;

k. now create password file to enable remore sysdba login

$ export ORACLE_SID=MYDB
$ orapwd file=$ORACLE_HOME/dbs/orapw password=oracle entries=5

l. create spfile and restart database

$ export ORACLE_SID=MYDB
$ sqlplus / as sysdba

SQL> create spfile from pfile='[location of the newly created pfile]'
SQL> shutdown immediate
SQL> startup

m. Check alert log for error and take a level 0 backup using DP

n. optionally enter a new entry for the database in "/etc/oratab", needed for configuration [dbca]


If there is a 32bit and 64bit difference between source and destination database, we may encounter below error on step "10.j" :
--------------------------------------------------------------------------------------
RMAN-06900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT row
RMAN-06901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rows
ORACLE error from target database:
ORA-00604: error occurred at recursive SQL level 1
ORA-06553: PLS-801: internal error [56327]
ORA-06553: PLS-801: internal error [56327]
--------------------------------------------------------------------------------------
Problem is that, backup originally was done from Oracle 64bit, but restored to Oracle 32bit.
Please use "MOS NOTE [ID 62290.1] — Changing between 32-bit and 64-bit Word Sizes" to resolve this issue.

Related Links:
http://www.orafusion.com/art_rman3.htm
http://yurijs.wordpress.com/2010/01/27/backup-issue/
ttp://download.oracle.com/docs/cd/B28359_01/backup.111/b28270/rcmcomre.htm

For HPDP Related Config:
http://razib2020.blogspot.com/2010/07/installation-of-dp-cell-manager-purging.html