Thursday, February 7, 2013

Oracle Password Verification Function

I am sharing a password verification function that will satisfy most of the common complexity requirements. I have modified default password verification function that is available in ORACLE_HOME by default.

  • It will enforce below:
  • Alphanumeric
  • At least 8 characters
  • Password should not contain username
  • At least one punctuation mark
  • Should be mixed case




CREATE OR REPLACE FUNCTION f_passwordVerification(p_username    IN VARCHAR2,
                                                 p_password    IN VARCHAR2,
                                                 p_oldPassword IN VARCHAR2)
  RETURN BOOLEAN IS

  l_password VARCHAR2(100) := UPPER(p_password);
  l_alpha    BOOLEAN := FALSE;
  l_numeric  BOOLEAN := FALSE;
  ispunct    BOOLEAN := FALSE;
  ismixed    BOOLEAN := FALSE;
  punctarray varchar2(25);
  invalidPassword EXCEPTION;

BEGIN

  ------------------------------------------
  -- Check for alpha and numeric characters.
  FOR j IN 1 .. LENGTH(p_password) LOOP
    IF INSTR('01234567890', SUBSTR(l_password, j, 1)) != 0 THEN
      l_numeric := TRUE;
    ELSIF INSTR('ABCDEFGHIJKLMNOPQRSTUVWXYZ', SUBSTR(l_password, j, 1)) != 0 THEN
      l_alpha := TRUE;
    END IF;
    EXIT WHEN l_alpha AND l_numeric;
  END LOOP;

  IF NOT (l_alpha AND l_numeric) THEN
     raise_application_error(-20020, 'Password should be alphanumeric!!');
  END IF;
  ------------------------------------------

  -- Check length is bewteen 8
  IF LENGTH(l_password) < 8 THEN
    raise_application_error(-20020, 'Password should have atleast 8 characters!!');
  END IF;
  ------------------------------------------

  -- Check length username in contained in password
  IF instr(upper(p_password), upper(p_username)) > 0 THEN
    raise_application_error(-20020, 'Password should not contain username!!');
  END IF;
  ------------------------------------------

  -- Check if the password contains at least one punctuation mark
  punctarray := '!"#$%&()``*+,-/:;<=>?_';

  ispunct := FALSE;
  FOR i IN 1 .. length(punctarray) LOOP
    FOR j IN 1 .. LENGTH(p_password) LOOP
      IF substr(p_password, j, 1) = substr(punctarray, i, 1) THEN
        ispunct := TRUE;
      END IF;
    END LOOP;
  END LOOP;
  IF ispunct = FALSE THEN
    raise_application_error(-20020, 'Password should contain at least one punctuation mark!!');
  END IF;

  ------------------------------------------
  -- check for mixed case
  ismixed := FALSE;

  FOR i IN 1 .. length(p_password) LOOP
    if INSTR('ABCDEFGHIJKLMNOPQRSTUVWXYZ', SUBSTR(p_password, i, 1)) != 0 then
      FOR j IN 1 .. length(p_password) LOOP
        if INSTR('abcdefghijklmnopqrstuvwxyz', SUBSTR(p_password, j, 1)) != 0 then
          ismixed := TRUE;
        end if;
      end loop;
    end if;
  end loop;

  IF ismixed = FALSE THEN
     raise_application_error(-20020, 'Password should be mixed case!!');    
  END IF;

  RETURN(TRUE);

--EXCEPTION
  --WHEN invalidPassword THEN
    --RETURN(FALSE);  
END;
/

Saturday, January 12, 2013

Install Oracle Cloud Control 12C in an Active/Passive Environment for High Availability


Sharing my action plan to Install Oracle Cloud Control 12C in an Active/Passive Environment for High Availability.So that one node will have the Database running and another will have OEM applications running, they will be able to fail over to each other's node.

Database: Oracle 11gR2 [11.2.0.3]
OS      : RHEL 5.7
Machine : Two machine with atleast two shared mountpoint on a Redhat Cluster.


Overview and Requirements
-------------------------
The following conditions must be met for Cloud Control to fail over to a different host:

    1. The installation must be done using a Virtual Host Name and an associated unique IP address.
    >>> DNS and CLuster IP from OEM App, make sure that DNS associated with CLuster IP of OEM APP can be resolved both way
    
    2. Install on a shared disk/volume which holds the binaries and the gc_inst directory.
    >>> Install OEM APP on shared disk [keeping all the ORACLE_HOMEs on shared storage associated in cluster]

    3. The Inventory location must failover to the surviving node.
    >>> keeping Inventory on shared storage associated in cluster

    4. The software owner and time zone parameters must be the same on all cluster member nodes that will host this Oracle Management Service (OMS)
    >>> create users with same userID and username. MAker sure to configure NTP and same correct timezone 

Setting Up Shared Inventory
---------------------------
Use the following steps to set up shared inventory:
    1. Create your new ORACLE_BASE directory on shared storage:
      $ mkdir /oemapp/data01/orabase/
      $ chown -R oracle:oinstall /oemapp/data01/orabase/

    2. Create the Oracle Inventory directory under the new ORACLE_BASE:
      $ mkdir /oemapp/data01/orabase/oraInventory

    3. Create the oraInst.loc file in shared location
      $ vi /oemapp/data01/orabase/oraInst.loc
      ##Enter the path information to the Oracle Inventory directory and specify the group of the software owner as the oinstall user
      inventory_loc=/oemapp/data01/orabase/oraInventory
      inst_group=oinstall

Installing the Software
-----------------------
Refer to the following steps when installing the software:
   
    1. Point to the inventory location file oraInst.loc (under the ORACLE_BASE in this case), as well as specifying the host name of the virtual group(DNS form oem apps in this case)
       and install the oracle binaries on shared storage:
      $ ./runInstaller -invPtrloc /oemapp/data01/orabase/oraInst.loc ORACLE_HOSTNAME=oemapp.robi.com.bd -debug

when for putting middleware home location put the below:

/oemapp/data01/OEMAPP/middleware

user:weblogic(oem4Robi)
nodemanager(oem4Robi)

drop the repository from oemdb by following command:
emca -deconfig dbcontrol db -repos drop -SYS_PWD oem4Robi -SYSMAN_PWD oem4Robi -HOST oemdb.robi.com.bd

mkdir -p /oemdb/data01/OEMDB/data/mgmt/

run the root script on current node first and switch over and then run from node 2:

/oemapp/data01/OEMAPP/middleware/oms/allroot.sh

    2. Install Oracle Management Services on cluster member Host1(oemapp in our case).

    3. Continue the remainder of the installation normally.

    4. Once completed, copy the files oraInst.loc and oratab to /etc on all cluster member hosts (Host2, Host3, ...)


Starting Up Services
--------------------
Ensure that you start your services in the proper order. Use the order listed below:

    1. Establish the IP address on the active node.

    2. Start the TNS listener (if it is part of the same failover group).

    3. Start the database (if it is part of the same failover group).

    4. Start Cloud Control using emctl start oms

    5. Test functionality


Configuring the Software Library
--------------------------------

The software library location must be accessed by all Management Services. The configuration of software library is not performed during installation and needs to be configured post-install using the Enterprise Manager Console:

    1. On the Enterprise Manager home page, from the Setup menu, select Provisioning and Patching, and then select Software Library.

    2. Click the Provisioning subtab.

    3. On the Provisioning page, click the Administration subtab.

    4. In the Software Library Configuration section, click Add to set the Software Library Directory Location to a shared storage that can be accessed by any host running the Management Service.
    
/oemapp/data01/OEMAPP/middleware/oms/bin/emctl start oms

This information is also available at: 

/oemapp/data01/OEMAPP/middleware/oms/install/setupinfo.txt

See below for information pertaining to your Enterprise Manager installation:


Use the following URL to access:
--------------------------------
1. Enterprise Manager Cloud Control URL: https://oemapp.robi.com.bd:7799/em
2. Admin Server URL: https://oemapp.robi.com.bd:7101/console

The following details need to be provided during the additional OMS install:

1. Admin Server Hostname: oemapp.robi.com.bd
2. Admin Server Port: 7101

 NOTE:
 An encryption key has been generated to encrypt sensitive data in the Management Repository. If this key is lost, all encrypted data in the Repository becomes unusable. Please run following command to backup the Management Service configuration including the emkey and keep the backup on another system:
 emctl exportconfig oms -dir  


http://docs.oracle.com/cd/E24628_01/doc.121/e24473/ha_setup.htm#CEGCCCDE

DBFS on Exadata, Normal Oracle Database and on Client


I prepared below action plan while installing DBFS on both exadata and on an external ETL machine to mount DBFS from exadata.


-- ---------------------------
-- serverside [if not exadata]
-- ---------------------------
1. Check to see if the "kernel-devel" package is installed.
#rpm -q kernel-devel
kernel-devel-2.6.18-128.el5
#If not, then install it

2. Determine the kernel directory.
# echo /usr/src/kernels/`uname -r`-`uname -p`
/usr/src/kernels/2.6.18-128.el5-x86_64

3. Install the FUSE package as the "root" user using the following commands, substituting your kernel 

directory.
--There may be WARNING regarding "--with-kernel",  just ignore it
--download fuse-2.7.3.tar.gz from "http://fuse.sourceforge.net/"
# tar -xzvf fuse-2.7.3.tar.gz
# cd fuse-2.7.3
# ./configure --prefix=/usr --with-kernel=/usr/src/kernels/2.6.18-128.el5-x86_64
# make
# make install
# /sbin/depmod
# /sbin/modprobe fuse
# chmod 666 /dev/fuse
# echo "/sbin/modprobe fuse" >> /etc/rc.modules
# chmod 744 /etc/rc.modules

4. install below rpms
fuse-libs-2.7.4-8.el5
fuse-devel-2.7.4-8.el5
fuse-2.7.4-8.el5

5. create DBFS tablespace
SQL> create bigfile tablespace tbs_dbfs datafile '+DATA2' size 200G autoextend off;

6. create DBFS user
SQL> create user dbfs_user identified by dbfs_passwd default tablespace tbs_dbfs quota unlimited on tbs_dbfs;
SQL> grant create session, create table, create procedure, dbfs_role to dbfs_user;

7. create the file system in tablespace 
cd $ORACLE_HOME/rdbms/admin
sqlplus dbfs_user/dbfs_passwd@DB

SQL> @dbfs_create_filesystem.sql tbs_dbfs my_dbfs  
--tbs_dbfs= DBFS TABLESPACE, my_dbfs=USER GIVEN NAME OF FILESYSTEM
[
-- for advanced setting use below command 
SQL> start dbfs_create_filesystem_advanced tbs_dbfs DBFS nocompress nodeduplicate noencrypt non-partition
]

[
--to drop a file system, this can be done using the "dbfs_drop_filesystem.sql" script with the file system 

name.
cd $ORACLE_HOME/rdbms/admin
sqlplus test/test

SQL> @dbfs_drop_filesystem.sql my_dbfs
--my_dbfs=USER GIVEN NAME OF FILESYSTEM which created in previous step
]

8. add oracle to fuse group
# groupadd fuse
# usermod -a -G fuse oracle

9. create dbfs mount point 
# mkdir /data01/DBFS_DIR
# chown oracle:oinstall /data01/DBFS_DIR

10. to let "dbfs os user(oracle)" provide access to other and root
# echo user_allow_other > /etc/fuse.conf -- to let "dbfs os user(oracle)" provide access to other and root 

11. using oracle user collect group id of oinstall group in the DB server 
$cat /etc/group|grep oinstall|cut -d: -f3 

12. using oracle user create a password file
$ echo dbfs_passwd > $HOME/passwd.txt
chmod 600 $HOME/passwd.txt #only oracle 

13. mount DBFS for only mounter
export ORACLE_HOME=;export TNS_ADMIN=;export 

LD_LIBRARY_PATH=$ORACLE_HOME/lib;$ORACLE_HOME/bin/dbfs_client dbfs_user@DEVDW  -o direct_io /data01/DBFS_DIR < 

$HOME/passwd.txt &
--or
-- for root also
export ORACLE_HOME=;export TNS_ADMIN=;export 

LD_LIBRARY_PATH=$ORACLE_HOME/lib;$ORACLE_HOME/bin/dbfs_client dbfs_user@DEVDW -o allow_root  -o direct_io 

/data01/DBFS_DIR < $HOME/passwd.txt &

14. unmount
fusermount -u /data01/DBFS_DIR

-- -------------------
-- IF DB is on Exadata 
-- -------------------

1. OS level COnfig
# usermod -a -G fuse oracle
# mkdir /u01/app/DBFS_DIR
# chown oracle:oinstall /u01/app/DBFS_DIR
# echo user_allow_other > /etc/fuse.conf

2. Create tablespace and user
create bigfile tablespace tbs_dbfs datafile '+DATA2' size 200G autoextend off;
create user dbfs_user identified by dbfs_passwd default tablespace tbs_dbfs quota unlimited on tbs_dbfs;
grant create session, create table, create procedure, dbfs_role to dbfs_user;
start dbfs_create_filesystem_advanced tbs_dbfs DBFS nocompress nodeduplicate noencrypt non-partition

3. create a file with password
$echo dbfs_passwd > $ORACLE_HOME/passwd.txt
#chown 700 /u01/app/oracle/product/11.2.0/dbhome_1/passwd.txt

4. Create mount script
export 

ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1;TNS_ADMIN=/u01/app/oracle/product/11.2.0/dbhome_1/network/

admin;export LD_LIBRARY_PATH=/u01/app/oracle/product/11.2.0/dbhome_1/lib;nohup 

/u01/app/oracle/product/11.2.0/dbhome_1/bin/dbfs_client dbfs_user@DEVDW -o allow_other,direct_io 

/u01/app/DBFS_DIR < /u01/app/oracle/product/11.2.0/dbhome_1/passwd.txt &
-- for root also
export 

ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1;TNS_ADMIN=/u01/app/oracle/product/11.2.0/dbhome_1/network/

admin;export LD_LIBRARY_PATH=/u01/app/oracle/product/11.2.0/dbhome_1/lib;
nohup /u01/app/oracle/product/11.2.0/dbhome_1/bin/dbfs_client dbfs_user@DEVDW -o allow_root,direct_io 

/u01/app/DBFS_DIR < /u01/app/oracle/product/11.2.0/dbhome_1/passwd.txt &

5. command for unlount
fusermount -u /u01/app/DBFS_DIR


-- -----------
-- client side
-- -----------


1. Check to see if the "kernel-devel" package is installed.
#rpm -q kernel-devel
kernel-devel-2.6.18-128.el5
#If not, then install it

2. Determine the kernel directory.
# echo /usr/src/kernels/`uname -r`-`uname -p`
/usr/src/kernels/2.6.18-128.el5-x86_64

3. Install the FUSE package as the "root" user using the following commands, substituting your kernel 

directory.
--There may be WARNING regarding "--with-kernel",  just ignore it
# tar -xzvf fuse-2.7.3.tar.gz
# cd fuse-2.7.3
# ./configure --prefix=/usr --with-kernel=/usr/src/kernels/2.6.18-128.el5-x86_64
# make
# make install
# /sbin/depmod
# /sbin/modprobe fuse
# chmod 666 /dev/fuse
# echo "/sbin/modprobe fuse" >> /etc/rc.modules
# chmod 744 /etc/rc.modules

4. install below rpms
fuse-libs-2.7.4-8.el5
fuse-devel-2.7.4-8.el5
fuse-2.7.4-8.el5

5. permit other user to execute dbfs_client
#chmod 755 <$ORACLE_HOME>/bin/dbfs_client

6. permit other user to read tnsnames.ora
#chmod 755 <$ORACLE_HOME>/network/admin/tnsnames.ora

7. create OS dbfsuser and group
#groupadd -g 101 fuse [with same ID as server]
#create user dbfs_user -g fuse

8. create DBFS mount point
#mkdir /data01/DBFS_DIR
#chown oracle:fuse /data01/DBFS_DIR
#chmod 775 /data01/DBFS_DIR

9. create password file for dbfs  
#echo dbfs_passwd > $HOME/passwd.txt
#chown oracle:fuse $HOME/passwd.txt
#chmod 640 $HOME/passwd.txt

10. mount DBFS for only mounter
export ORACLE_HOME=;export TNS_ADMIN=;export 

LD_LIBRARY_PATH=$ORACLE_HOME/lib;$ORACLE_HOME/bin/dbfs_client dbfs_user@DEVDW  -o direct_io /data01/DBFS_DIR < 

$HOME/passwd.txt &
--or
-- for root also
export ORACLE_HOME=;export TNS_ADMIN=;export 

LD_LIBRARY_PATH=$ORACLE_HOME/lib;$ORACLE_HOME/bin/dbfs_client dbfs_user@DEVDW -o allow_root  -o direct_io 

/data01/DBFS_DIR < $HOME/passwd.txt &

11. with DBFS mounted create directory fully accessible by both client user & server group
#mkdir /data01/DBFS_DIR//EXT_DIR/
#c€hown dbfs_user: /data01/DBFS_DIR/EXT_DIR  # here 

owner=local user, group = DBFS DB oinstall group

12. unmount
fusermount -u /data01/DBFS_DIR



Rename/Change Location Datafile in ASM


To renamee/change location a datafile in ASM I followed below steps:

*** Considering database in archivelog mode

1. File the target file to rename.
select * from dba_data_files where tablespace_name='TBS_DW' order by file_name desc;

2. Put the file to offline from RMAN
SQL> ALTER DATABASE DATAFILE '+SYSTEMDG/oradb/datafile/tbs_dw.276.780864751' OFFLINE;

3. Execute copy from RAMN to the new location
RMAN> 
run{
allocate channel c1 device type disk;
COPY DATAFILE '+SYSTEMDG/oradb/datafile/tbs_dw.276.780864751' TO '+DATA1';
release channel c1;
}
---------- OUTPUT ---------
Starting backup at 17-APR-12
channel c1: starting datafile copy
input datafile file number=00631 name=+SYSTEMDG/oradb/datafile/tbs_dw.276.780864751
output file name=+DATA1/oradb/datafile/tbs_dw.915.780866609 tag=TAG20120417T192328 RECID=3 STAMP=780870208 --<<< use name of output file in the next rename stage
channel c1: datafile copy complete, elapsed time: 00:00:25
Finished backup at 17-APR-12
---------------------------

4. Rename the file from RMAN
RMAN> run {
allocate channel c1 device type disk;
set newname for datafile '+SYSTEMDG/oradb/datafile/tbs_dw.276.780864751' to '+DATA1/oradb/datafile/tbs_dw.915.780866609';
switch datafile all;
release channel c1;

---------- OUTPUT ---------
SQL> RECOVER DATAFILE '+DATA1/oradb/datafile/tbs_dw.915.780866609';
ORA-00279: change 3110310780083 generated at 04/17/2012 19:42:29 needed for
thread 8
ORA-00289: suggestion :
+RECO/oradb/archivelog/2012_04_17/thread_8_seq_692.1501.780864785
ORA-00280: change 3110310780083 for thread 8 is in sequence #692

Specify log: {=suggested | filename | AUTO | CANCEL}  -----<<< just enter
+RECO/oradb/archivelog/2012_04_17/thread_8_seq_692.1501.780864785
---------------------------

5. make the copied file online
SQL> ALTER DATABASE DATAFILE '+DATA1/oradb/datafile/tbs_dw.915.780866609' ONLINE;

6. check the file names again.
select * from dba_data_files where tablespace_name='TBS_DW' order by file_name desc;

7. drop using the old name
ASM: SQL> ALTER DISKGROUP ASMDSK2 DROP FILE tbs_dw.276.780864751;

Exadata Storage Cell Disk Change


I follow below steps while changing and faulty disk on Exadata. We are using Exadata storage SW version 11.2.2.4.0.

1. You have identified following disk to be replaced:
       
$ cellcli -e list physicaldisk detail
         name:                   18:4
         deviceId:               10
         diskType:               HardDisk
         enclosureDeviceId:      18
         errMediaCount:          471
         errOtherCount:          17
         foreignState:           false
         luns:                   0_4
         makeModel:              "SEAGATE ST360057SSUN600G"
         physicalFirmware:       0805
         physicalInsertTime:     2010-03-26T05:05:46+06:00
         physicalInterface:      sas
         physicalSerial:         E08WD8
         physicalSize:           558.9109999993816G
         slotNumber:             4
         status:                 critical

2. You need to make sure that there is no other disks are offline/missing in other cells(failgroup) :

select count(*),failgroup from gv$asm_disk group by failgroup;


3. Check ASM_POWER_LIMIT and if it is 0 then set it to non zero value so that after the disk replacement rebalance can happen and the newly added disks will have data written on them.
This parameter can be set to any value between 0 to 11.
--on asm instance
alter system set asm_power_limit=10;

4. Make sure there are no rebalance operation running on any instance on any node:

select * from gv$asm_operation;

5. Identify the ASM disks which are existing on the physical disk to be replaced:

a. On the cell node execute following:

- Find the lun number "0_11" in the list physicaldisk detail output
# cellcli -e list physicaldisk detail
.....
luns: 0_4
.....

- Find the celldisk created for this lun (physicaldisk)
# cellcli -e list celldisk where lun=0_4
         CD_04_axdwcel06         not present

- Find the griddisks created for this celldisk
# cellcli -e list griddisk where celldisk=CD_04_axdwcel06
         DATA1_CD_04_axdwcel06           not present
         DATA2_CD_04_axdwcel06           not present
         RECO_CD_04_axdwcel06            not present
         SYSTEMDG_CD_04_axdwcel06        not present



b. Execute following on any db node connecting to the ASM instance:

SQL> select name, substr(path,1,50) , mount_status, header_status, state from gv$asm_disk where name in ('DATA1_CD_04_AXDWCEL06','DATA2_CD_04_AXDWCEL06','SYSTEMDG_CD_04_AXDWCEL06','RECO_CD_04_AXDWCEL06');

NAME SUBSTR           (PATH,1,50)                        MOUNT_S HEADER_STA STATE
------------------------------ ----------------------------------------   ------
DATA_CD_11_DMORLCEL02 o/192.168.10.6/DATA_CD_11_dmorlcel02 CACHED  MEMBER  NORMAL
RECO_CD_11_DMORLCEL02 o/192.168.10.6/RECO_CD_11_dmorlcel02 CACHED  MEMBER  NORMAL



6. Now drop the disk from the diskgroup(s):

alter diskgroup DATA1 drop disk 'DATA1_CD_04_AXDWCEL06';
alter diskgroup DATA2 drop disk 'DATA2_CD_04_AXDWCEL06';
alter diskgroup SYSTEMDG drop disk 'SYSTEMDG_CD_04_AXDWCEL06';
alter diskgroup RECO drop disk 'RECO_CD_04_AXDWCEL06';

7. Wait until the Oracle ASM disks associated with the grid disks on the bad disk have been successfully dropped by querying the V$ASM_DISK_STAT view.

V$ASM_DISK_STAT displays disk statistics in the same way that V$ASM_DISK does, but without performing discovery of new disks. This results in a less expensive operation. However, since discovery is not performed, the output of this view does not include any data about disks that are new to the system.


8. Mark this disk at the hardware level to be serviced:

cellcli -e 'alter physicaldisk 18:4 serviceled on'

Here disk_name should be replaced by the name of the physical disk '18:4'.

This step will help in identifying the disk to be replaced at the h/w level. Please note that this command may  not work in version less than 11.2.2.2. but it is not mandatory. it just helps in identifying the disk to be replaced.

9. Now replace the disk at the h/w level. The service LED will glow for this disk as done in the previous step.

10. After replacing the disk Exadata s/w will automatically create celldisk, griddisks. You can tail the cell alert.log and ASM alert.log to see this activity. But the disks will not be added back to the ASM as we did not use FORCE option while dropping them. So we need to add them manually:

alter diskgroup DATA1 add  disk 'o/10.10.0.10/DATA1_CD_04_AXDWCEL06';
alter diskgroup DATA2 add  disk 'o/10.10.0.10/DATA2_CD_04_AXDWCEL06';
alter diskgroup RECO add  disk 'o/10.10.0.10/RECO_CD_04_AXDWCEL06';
alter diskgroup SYSTEMDG add  disk 'o/10.10.0.10/SYSTEMDG_CD_04_AXDWCEL06';

-- Location of the cell alert.log file:
/opt/oracle/cell11.2.2.4.0_LINUX.X64_110929/log/diag/asm/cell/axdwcel06/trace/alert.log

-- Location of the ASM alert.log
/u01/app/oracle/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log


11. Check until the rebalance completes using following sql:
select * from gv$asm_operation;

12. Once the rebalance is over check the status of the ASM disks again and verify that everything is fine:

SQL> select name, substr(path,1,50) , mount_status, header_status, state from gv$asm_disk where name in ('DATA1_CD_04_AXDWCEL06','DATA2_CD_04_AXDWCEL06','SYSTEMDG_CD_04_AXDWCEL06','RECO_CD_04_AXDWCEL06');

select * from gv$asm_disk

13. Verify all grid disks have been successfully put online using the following command from the cell where disk change is done:
    (Wait until asmmodestatus is ONLINE for all grid disks
      and Oracle ASM synchronization is only complete when all grid disks show asmmodestatus=ONLINE.)
     cellcli -e list griddisk attributes name,asmmodestatus,asmdeactivationoutcome

14. also check the alert history from the cell where disk change is done
cellcli -e list alerthistory

Shutting Down Exadata Storage Cell


Normally I follow below steps while shutting down an Exadata Storage Cell for maintenance purpose. (We were using Exadata storage SW version 11.2.2.4.0.) 


1. check disk repair time from ASM instance and ajust it if required.
select dg.name, a.value
  from v$asm_diskgroup dg, v$asm_attribute a
 where dg.group_number = a.group_number
   and a.name = 'disk_repair_time';

2. from both ASM & DB instance, make sure that all the DISKs are online

select header_status,mode_status,state,failgroup,count(*) from gv$asm_disk group by header_status,mode_status,state,failgroup;

3. from ASM instanvce wait until below query returns zero rows
select * from gv$asm_operation;

4. Next you will need to check if ASM will be OK if the grid disks go OFFLINE. The following command should return 'Yes' for the grid disks being listed:
   cellcli -e list griddisk attributes name,asmmodestatus,asmdeactivationoutcome 


5. inactivated all griddisks
cellcli -e ALTER GRIDDISK ALL INACTIVE

6. Confirm that the griddisks are now offline by performing the following actions:
   a. Execute the command below and the output should show asmmodestatus=UNUSED and asmdeactivationoutcome=Yes for all griddisks once the disks are offline in ASM. Only then is it safe to proceed with shutting down or restarting the cell:
      cellcli -e list griddisk attributes name,asmmodestatus,asmdeactivationoutcome
   b. List the griddisk to confirm that all show inactive:
      cellcli -e LIST GRIDDISK    

7. from both ASM & DB instance, make sure that all the DISKs are online

select header_status,mode_status,state,failgroup,count(*) from gv$asm_disk group by header_status,mode_status,state,failgroup;     

8. The following command will reboot Oracle Exadata Storage Server immediately:
   (When powering off Oracle Exadata Storage Servers, all storage services are automatically stopped.)
        # shutdown -y now   
        sync 
        sync
        init 0   

9. check validation log's last 20 lines all should bew PASSED or NOHUP RUN or BACKGROUND RUN
tail -20 /var/log/cellos/validations.log

10. Once the cell comes back online - you will need to reactive the griddisks:

        cellcli -e alter griddisk all active

11. Issue the command below and all disks should show 'active':

        cellcli -e list griddisk

12. Verify all grid disks have been successfully put online using the following command:
    (Wait until asmmodestatus is ONLINE for all grid disks 
      and Oracle ASM synchronization is only complete when all grid disks show asmmodestatus=ONLINE.)
     cellcli -e list griddisk attributes name,asmmodestatus,asmdeactivationoutcome

13. from both ASM & DB instance, make sure that all the DISKs are online

select header_status,mode_status,state,failgroup,count(*) from gv$asm_disk group by header_status,mode_status,state,failgroup;

14. Next you will need to check if ASM will be OK if the grid disks go OFFLINE. The following command should return 'Yes' for the grid disks being listed:
   cellcli -e list griddisk attributes name,asmmodestatus,asmdeactivationoutcome 
-- >>> check that all the grid disks are online

15. from ASM instanvce wait until below query returns zero rows
select * from gv$asm_operation;

16. check cell alert for any uncleared critical alerts
cellcli -e list alerthistory

17. check below from both DB & ASM 
select * from gv$instance;

Shutting Down a Exadata DB Node [same on normal RAC]


While we need to shutdown any of exadata(same for any RAC DB) DB Nodes I normally follow below checklist. 

1. from DB & ASM node run below query and cahek all the nodes are in open staus
select * from gv$instance;
2. login to the target DB node with oracle user and set .oracrs.sh env

3. now shutdown the instance on the target node using below command
/u01/app/11.2.0/grid/bin/srvctl stop instance -d ORADB -i

4. FROM DB node run belwo quert to be sure that target instance is now down
select * from gv$instance;

5. now login as root and run below command to shudown CRS
/u01/app/11.2.0/grid/bin/crsctl stop crs

-- from DB node run below command and make sure that crs is down [CRS-0184]
/u01/app/11.2.0/grid/bin/crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.

6. FROM ASM run below query to make sure that ASM on tgaret hots is down
select * from gv$instance;

7. shutdown the DB machine

8. startup the DB machine

9. check with below command the all the cluster services are online except .gsd and .oc4j
/u01/app/11.2.0/grid/bin/crs_stat -t|grep OFFLINE
--from DB node with oracle user and .oracrs.sh env run below command to make sure that all the ASM disks are online
asmcmd
ASMCMD> ls

. oracrs.sh
sqlplus / as sysasm

10. if any disk is missing the from ASM instance of the target machine run below command to make the perticular disk online
alter diskgroup mount;

11. chek that DB instance is online or not if not automatically online use below command to make it online
srvctl start -d axdw -i

12. from ASM & DB instance user below query to check all the ASM and DB instances are online
select * from gv$instance;

13. from ASM & DB instance user below query to check all the ASM DISKS are online from all the instance
select header_status,mode_status,state,failgroup,count(*) from gv$asm_disk group by header_status,mode_status,state,failgroup;

14. check alertlog of both db & asm for the target node