Monday, January 19, 2015

Preparing Hosts(VMs) to Install Cloudera Hadoop (Install Cloudera Hadoop part 1 of 2):

Here I am sharing my activity cookbook I followed to install a development environment for Cloudera Hadoop cluster using Virtual Box. Using almost same method you can easily install the Cloudera Hadoop on production.

Here is the part 1 of 2, hosts/nodes preparation. For Cloudera Manager & Cloudera hadoop installation please check part 2.


1. Target:

Hadoop:
Hadoop Cluster with below:
- 1 management node: acting as GW to the cluster and hasting Cloudera manager with 4G ram, 2 vcore, 25G local disk
- 1 name node: acting as primary namenode and standby resource manager with 2G ram, 1 vcore, 25G local disk 
- 1 resource manager node: acting as promary resource manager and standby namenode with 2G ram, 1 vcore, 25G local disk
- 3 datanodes: datanodes having all hadoop process and having 3 disk volumes each. with 2G ram, 1 vcore, 25G local disk
- Onlt management node will have internet connection, other 5 not. It is to emulate production data center environment.

VMs for hadoop:
Configure 1 virtual guest1 with:
- can ssh from host
- can connect to internet
- can interact with other guest on the same host
- having 25GB local storage
- having 1 GB ram
- 1 virtual CPU

2. Target Virtual machine:
We used virtual box as our virtualization SW.
Make sure virtualization support is activated for the host. If not enabled plese enable it from BIOS.

3. Create a virtual hostwith below network config:
Adapter 1: hostonly (on eth0)
Adapter 2: NAT (on eth1)

4. Install CenOS/RHEL with required partitions. For our case we are using only below 3 partitions. It will help use all available space the space:
- /
- /boot
- /home

5. Configure network as below:

a. On the host Hostonly Network(eth0):
- On Virtual box guest configuration: The default IP for hostonly virtual interface on the host machine is 192.168.56.1 and no gateway, keep it untouched.

- On eth0 configuration file:
- Keep DEVICE, HWADDR & UUID untouch. 
- Chnage ONBOOT=yes
- Change BOOTPROTO to PROTO put value "static"
- add IPADDR=
- no gateway no other things. 
- If any just remove them
- this interface will be used to communicate with the host and other guests

- For exaple, ifcfg-eth0 file looks like below, for other machine only  IPADDR will be changed:
[root@base ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
HWADDR=08:00:27:C6:2A:31
TYPE=Ethernet
UUID=e8e7aafe-2033-4cc5-ae93-bea49b5b3528
ONBOOT=yes
PROTO=static
IPADDR=192.168.56.201
[root@base ~]#

b. On NAT (eth1):
- change ONBOOT=yes 
- add BOOTPROTO=dhcp (if not there)
- keep rest untouched
- this will be used to access internet
- For example ifcfg-eth1 will look like below:
[root@base ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
HWADDR=08:00:27:AC:01:6D
TYPE=Ethernet
UUID=5ba5cf93-7526-4acb-88d9-9ce14439df80
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=dhcp

c. Change your host name change as below:
[root@base ~]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=base.example.com
[root@base ~]#


d. NO need to do any change on DNS (resolv.conf)

e. (Optional) Set the swappiness to 0 to avoid swaping, as per the recommendation of Cloudera manager:
# vi /etc/sysctl.conf
vm.swappiness = 0
# sysctl -p
# cat /proc/sys/vm/swappiness
0

Ref: https://blogs.oracle.com/fatbloke/entry/networking_in_virtualbox1

6. Disable selinux
vi /etc/selinux/config
SELINUX=disabled

7. Disable IPV6 by issuing below commands as root:
# vi /etc/sysctl.conf and add below two lines:
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.all.disable_ipv6 = 1

then run "sysctl -p" as root
# sysctl -p

9. diable firewall
chkconfig iptables off

10. vi /etc/yum/pluginconf.d/fastestmirror.conf
enabled=0

11. reboot and check all changes are persistant


12. Setup SSH

To also simplify the access between hosts, install and setup SSH keys and defined them as already authorized
- do below on the base node. While we will cloning this base to create orger nodes it will be there. No need to copy again.

$ yum -y install perl openssh-clients
$ ssh-keygen (type enter, enter, enter)
$ cd ~/.ssh
$ cp id_rsa.pub authorized_keys

- Modify the ssh configuration file. Uncomment the following line and change the value to no; this will prevent the question when connecting with SSH to the host.

# vi /etc/ssh/ssh_config
StrictHostKeyChecking no

13. edit /etc/hosts file as per tour need: 
vi /etc/hosts
192.168.56.201 dvhdmgt1.example.com dvhdmgt1  # Management node hosting Cloudera Manager
192.168.56.202 dvhdnn1.example.com dvhdnn1 # Name node
192.168.56.203 dvhdjt1.example.com dvhdjt1 # Jobtracker/Resource Manager
192.168.56.101 dvhddn01.example.com dvhddn01 # Datanode1
192.168.56.102 dvhddn02.example.com dvhddn02 # Datanode2
192.168.56.103 dvhddn03.example.com dvhddn03 # Datanode3

14. Clone the base with considering below for each of the nodes:

a. if NIC not up, then select both of adaptar and for both from advance options, refresh the MAC update HWADDD with corresponding MAC in eth0 & eth1 and reboot

b. When the system is up and both NICs (eth0 & eth1) are up do as below:
- open "/etc/udev/rules.d/*-persistent-net.rules" and check which MAC (ATTR{address}==) matched with which eth0/eth1 and rename to corresponding eth0/eth1 then remove/commentout other line if having eth0/eth1 not maching MAC
[ref: http://xmodulo.com/2013/04/how-to-clone-or-copy-virtual-machine-on-virtualbox.html]
c. Assign IP & MAC for corresponding nodes on eth0 only, Change only MAC for eth1 on file /etc/sysconfig/network-scripts/ifcfg-eth0 & 1
d. change the hostname on /etc/sysconfig/network as per point 13.
e. Reboot to make it effective and check
f. Following above steps create 6 VMs and configure CPU & RAM as listed in step 1.  
g. Except management node (dvhdmgt1), shutdown the NAT network(ifdown eth1) and set "ONBOOT=no" in file /etc/sysconfig/network-scripts/ifcfg-eth1. 

  
15. Mount jbod (only for data disks)
a. From virtual box assign 3 disks to each of the 3 datanodes.
b. Mount your data disks with noatime (e.g. /dev/sdc1 /mnt/disk3 ext4 defaults,noatime 1 2 which btw. implies nodiratime)
c. (Optionally) By default 5% of a HDD are reserved in ext filesystems for  critical processes can still write some data when the disk is full. (check by running tune2fs -l /dev/sdc1 and look at the Reserved block count). Down it to 1% by running: tune2fs -m 1 on all your data disks (i.e. tune2fs -m 1 /dev/sdc1)
http://blog.cloudera.com/blog/2014/01/how-to-create-a-simple-hadoop-cluster-with-virtualbox/

For eample do as below for each data nodes:
- # shutdown
- add 3 virtual storage HDD on data nodes
- start the machine
- use "fdisk -l" to check unformatted disks (should not be included in partition tables)
- partition each of whole disks as a single primary partitions with partition number "1" for all using fdisk:
# fdisk /dev/sdb then
then follow all the steps to make desired partitions(typically n>p>1>enter>enter>w) 
After that all the device will have partition with additiinal 1 in the corresponding device name(like, /dev/sdb will have /dev/sdb1)
- Format disks with ext4 
mkfs.ext4 /dev/sdb1
mkfs.ext4 /dev/sdc1
mkfs.ext4 /dev/sdd1
- (optionally) tune as per point c above:
tune2fs -m 1 /dev/sdb1
tune2fs -m 1 /dev/sdc1
tune2fs -m 1 /dev/sdd1

tune2fs -l /dev/sdb1 |grep "Reserved block count:"
tune2fs -l /dev/sdc1 |grep "Reserved block count:"
tune2fs -l /dev/sdd1 |grep "Reserved block count:"
- Mount data partitions
# mkdir -p /data/01
# mkdir -p /data/02
# mkdir -p /data/03

   # vi /etc/fstab
/dev/sdb1 /data/01 ext4 defaults,noatime 1 2
/dev/sdc1 /data/02 ext4 defaults,noatime 1 2
/dev/sdd1 /data/03 ext4 defaults,noatime 1 2

# mount -a


16. Prepare Cloudera Manager (CM) server node(on dvhdmgt1) as a proxy for yum(as only CM will have internet connection):

16.1 install squid and enable local caching:
# yum install squid
16.2 specify the caching directory (here we are caching about 7000 MB):
# grep cache_dir /etc/squid/squid.conf
#cache_dir ufs /var/spool/squid 100 16 256
cache_dir ufs /var/spool/squid 7000 16 256

16.3 You will also have to allow connections to port 3128 or stop the firewall. For our case it is not running.
16.4 start squid server (on CM node):
#service squid start
init_cache_dir /var/spool/squid... Starting squid: .       [  OK  ]
 
16.5 Add squid to chkconfig:
# chkconfig squid on
# chkconfig --list squid
squid           0:off   1:off   2:on    3:on    4:on    5:on    6:off

17. Create repo file for cloudera manager with proper version (on CM node):

17.1 Cloudera recommends installing products using package management tools such as yum for Red Hat compatible systems. We will follow this also for Redhat systems.
17.2  (on CM node) download repo file "http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo" and copy to the "/etc/yum.repos.d/" directory.
17.3  (on CM node) Edit the file to change the baseurl to point to the specific version of Cloudera Manager you want to download. For us, we want to install Cloudera Manager version 5.0.2. So our final "/etc/yum.repos.d/cloudera-manager.repo" file will be as below.

[cloudera-manager]
# Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64            
name=Cloudera Manager
baseurl=http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.0.2/
gpgkey = http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera    
gpgcheck = 1

17.4 do above on CM node(dvhdmgt1) then distribute the repo file to all nodes
[root@dvhdmgt1 yum.repos.d]# scp cloudera-manager.repo dvhdnn1:/etc/yum.repos.d
[root@dvhdmgt1 yum.repos.d]# scp cloudera-manager.repo dvhdjt1:/etc/yum.repos.d
[root@dvhdmgt1 yum.repos.d]# scp cloudera-manager.repo dvhddn01:/etc/yum.repos.d
[root@dvhdmgt1 yum.repos.d]# scp cloudera-manager.repo dvhddn02:/etc/yum.repos.d
[root@dvhdmgt1 yum.repos.d]# scp cloudera-manager.repo dvhddn03:/etc/yum.repos.d


18. Point yum proxy to the CM node(dvhdmgt1) (on all node except dvhdmgt1):
18.1 On all the servers that need to use the cache, set the proxy configuration in their /etc/yum.conf file to be the cache server on port 3128.
18.2 for our case we will use CM server IP:
# grep proxy /etc/yum.conf
 proxy=http://192.168.56.201:3128
18.3 test with "yum info jdk", it should be successfull to load info From repo "cloudera-manager" as per set(step 17 above) in CM node:
# yum info jdk
base                                                                                                                                             | 3.7 kB     00:00
base/primary_db                                                                                                                                  | 4.6 MB     00:01
extras                                                                                                                                           | 3.3 kB     00:00
extras/primary_db                                                                                                                                |  19 kB     00:00
updates                                                                                                                                          | 3.4 kB     00:00
updates/primary_db                                                                                                                               | 171 kB     00:00
Installed Packages
Name        : jdk
Arch        : x86_64
Epoch       : 2000
Version     : 1.6.0_31
Release     : fcs
Size        : 143 M
Repo        : installed
From repo   : cloudera-manager
Summary     : Java(TM) Platform Standard Edition Development Kit
URL         : http://java.sun.com/
License     : Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved. Also under other license(s) as shown at the Description field.
Description : The Java Platform Standard Edition Development Kit (JDK) includes both
: the runtime environment (Java virtual machine, the Java platform classes
: and supporting files) and development tools (compilers, debuggers,
: tool libraries and other tools).
:
: The JDK is a development environment for building applications, applets
: and components that can be deployed with the Java Platform Standard
: Edition Runtime Environment.


# yum list available|grep -i cloudera-manager
cloudera-manager-server.x86_64            5.0.2-1.cm502.p0.297.el6       cloudera-manager
cloudera-manager-server-db-2.x86_64       5.0.2-1.cm502.p0.297.el6       cloudera-manager
enterprise-debuginfo.x86_64               5.0.2-1.cm502.p0.297.el6       cloudera-manager


19. At this point all the VM hosts are ready to install Cloudera Hadoop.

20. Please follow next post on install Cloudera Manager

Thursday, November 6, 2014

OLTP vs MPP vs Hadoop


Some of my friends asked me about OLTP, MPP and Hadoop. I tried to expain them as below.
This is related to the time writing this post. Things are changing so fast :).

OLTP Databases (Oracle,DB2) vs MPP (Netezza, Teradata, Vertica et.):

                1. - DB Oracle or DB2 needs to read data from disk to memory before start processing, so very fast in memory calculation.
                   - MPP takes the processing as close possible to the data, so less data movement
             
                2. - DB Oracle or DB2 is good for smaller OLTP (transaction) operations. It also maintains very high level of data intigrity.
                   - MPP is good for batch processing. Some of the MPP(Netezza, Vertica) overlooks intigrity like enforcing unique key for the sake of batch performance.


Hadoop(without impala or EMC HAWQ) vs MPP:

                1. - Conventional MPP database stores data in a matured internal structure. So data loading and data processing with SQL is efficient.
                   - There are no such structured architecture for data stored on hadoop. So, accessing and loading data is not as efficient as conventional MPP systems.
                2. - With conventional MPP, it support only relational models(row-column)
                   - hadoop support virtually any kind of data.
             
                * However the main objective of MPP and hadoop is same, process data parallely near storage.
             

Cloudera impala(or pivotal HAWQ) vs MPP:

                1. - MPP supports advanced indatabase analytics
                   - Till now (impala 2.0) started supporting "SQL 2003" which may lead them to intruduce indatabase analytics.
                2. - MPP databases have industry standard security features and well defined user schema structure.
                   - Impala has very immatured security system and virtually no user schema.
                3. - MPP only supports only vendor specific filesystem and need to load data using specific loading tool.
                   - impala supports most open file formats (text, parquate)
             
                * However impala seems to become a MPP & Columnar like Vertica but cheap & open database system in near future. Just need to implement security and advance indatabase analytics.


How to choice what (in general and my personal opinion):


1. OLTP Databases (Oracle,DB2, MySQL, MS SQL, Exadata):
                - Transaction based application
                - Smaller DWH
                * However Exadata is a hybrid system and I have experience to handle DWH with ~20TB data.

2. MPP (Netezza, Teradata, Vertica)
                - Bigger Data warehouse (may be having tables with size more than 4-5 TB)  
                - Needs no or little pre-processing
                - Needs faster batch processing speed
                - In database analytics

3. Only Hadoop:
                - All data as heavily unstructured (documents, audio, video etc)
                - need to process in batch

4. Hadoop and using mainly Impala (or EMC HAWQ)
                - Need to have a DWH with low cost
                - No need to have advance analytics features
                - Can utilize open source tools
                - Not concern about security or limited number of users
             
5. Hadoop(with impala or HAWQ) + MPP:
                - Some Data need heavy pre-processing before ready to advance analytics.
                - Need cheaper query able archive or backup for older data


Referencs:
http://www.quora.com/Is-Impala-aiming-to-be-an-open-source-alternative-to-existing-MPP-solutions
http://www.quora.com/What-features-of-a-relational-database-are-most-useful-when-building-a-data-warehouse
http://blog.pivotal.io/big-data-pivotal/products/exploratory-data-science-when-to-use-an-mpp-database-sql-on-hadoop-or-map-reduc

Tuesday, September 10, 2013

Netezza: List Skew Tables Using Query

Below will will find out tables having data slices with "data slices contain more than 3 time of perfectly distributed data for a table"

In the bellow query I have used "3*((cast(100 as decimal(5))/(select count(*) from _V_DUAL_DSLICE)))" when means below:
"3 * 100 / total number of data slices", that is 3 times of a 100% data that is distributed evenly on all the available data slices.


select database_name,table_name,dsid,ALLOCATED_BYTES,USED_BYTES,PCT_DATA_IN_DSID 
from 
(select  the_table.database as database_name,
the_table.objname table_name,
        storage.dsid,
storage.ALLOCATED_BYTES,
storage.USED_BYTES,
storage.ALLOCATED_BYTES*100/sum(ALLOCATED_BYTES) over (partition by database_name,table_name) PCT_DATA_IN_DSID -- % of total size in this data slice
  FROM  _V_OBJ_RELATION_XDB the_table LEFT OUTER JOIN _V_SYS_OBJECT_DSLICE_INFO storage 
        on (  the_table.objid = storage.tblid  and storage.tblid > 200000)) TBL
  where --DATABASE_NAME = '' and -- if want to specify database name
PCT_DATA_IN_DSID > 3*((cast(100 as decimal(5))/(select count(*) from _V_DUAL_DSLICE)))  -- data slices contain more than 3 time of perfectly distributed data for a table
and ALLOCATED_BYTES > 100*1024*1024 --100 MB
--and table_name = '' --if want to specify tables name

--and DSID=630 --if want to specify Dataslice
  order by PCT_DATA_IN_DSID;

Thursday, February 7, 2013

Oracle Password Verification Function

I am sharing a password verification function that will satisfy most of the common complexity requirements. I have modified default password verification function that is available in ORACLE_HOME by default.

  • It will enforce below:
  • Alphanumeric
  • At least 8 characters
  • Password should not contain username
  • At least one punctuation mark
  • Should be mixed case




CREATE OR REPLACE FUNCTION f_passwordVerification(p_username    IN VARCHAR2,
                                                 p_password    IN VARCHAR2,
                                                 p_oldPassword IN VARCHAR2)
  RETURN BOOLEAN IS

  l_password VARCHAR2(100) := UPPER(p_password);
  l_alpha    BOOLEAN := FALSE;
  l_numeric  BOOLEAN := FALSE;
  ispunct    BOOLEAN := FALSE;
  ismixed    BOOLEAN := FALSE;
  punctarray varchar2(25);
  invalidPassword EXCEPTION;

BEGIN

  ------------------------------------------
  -- Check for alpha and numeric characters.
  FOR j IN 1 .. LENGTH(p_password) LOOP
    IF INSTR('01234567890', SUBSTR(l_password, j, 1)) != 0 THEN
      l_numeric := TRUE;
    ELSIF INSTR('ABCDEFGHIJKLMNOPQRSTUVWXYZ', SUBSTR(l_password, j, 1)) != 0 THEN
      l_alpha := TRUE;
    END IF;
    EXIT WHEN l_alpha AND l_numeric;
  END LOOP;

  IF NOT (l_alpha AND l_numeric) THEN
     raise_application_error(-20020, 'Password should be alphanumeric!!');
  END IF;
  ------------------------------------------

  -- Check length is bewteen 8
  IF LENGTH(l_password) < 8 THEN
    raise_application_error(-20020, 'Password should have atleast 8 characters!!');
  END IF;
  ------------------------------------------

  -- Check length username in contained in password
  IF instr(upper(p_password), upper(p_username)) > 0 THEN
    raise_application_error(-20020, 'Password should not contain username!!');
  END IF;
  ------------------------------------------

  -- Check if the password contains at least one punctuation mark
  punctarray := '!"#$%&()``*+,-/:;<=>?_';

  ispunct := FALSE;
  FOR i IN 1 .. length(punctarray) LOOP
    FOR j IN 1 .. LENGTH(p_password) LOOP
      IF substr(p_password, j, 1) = substr(punctarray, i, 1) THEN
        ispunct := TRUE;
      END IF;
    END LOOP;
  END LOOP;
  IF ispunct = FALSE THEN
    raise_application_error(-20020, 'Password should contain at least one punctuation mark!!');
  END IF;

  ------------------------------------------
  -- check for mixed case
  ismixed := FALSE;

  FOR i IN 1 .. length(p_password) LOOP
    if INSTR('ABCDEFGHIJKLMNOPQRSTUVWXYZ', SUBSTR(p_password, i, 1)) != 0 then
      FOR j IN 1 .. length(p_password) LOOP
        if INSTR('abcdefghijklmnopqrstuvwxyz', SUBSTR(p_password, j, 1)) != 0 then
          ismixed := TRUE;
        end if;
      end loop;
    end if;
  end loop;

  IF ismixed = FALSE THEN
     raise_application_error(-20020, 'Password should be mixed case!!');    
  END IF;

  RETURN(TRUE);

--EXCEPTION
  --WHEN invalidPassword THEN
    --RETURN(FALSE);  
END;
/

Saturday, January 12, 2013

Install Oracle Cloud Control 12C in an Active/Passive Environment for High Availability


Sharing my action plan to Install Oracle Cloud Control 12C in an Active/Passive Environment for High Availability.So that one node will have the Database running and another will have OEM applications running, they will be able to fail over to each other's node.

Database: Oracle 11gR2 [11.2.0.3]
OS      : RHEL 5.7
Machine : Two machine with atleast two shared mountpoint on a Redhat Cluster.


Overview and Requirements
-------------------------
The following conditions must be met for Cloud Control to fail over to a different host:

    1. The installation must be done using a Virtual Host Name and an associated unique IP address.
    >>> DNS and CLuster IP from OEM App, make sure that DNS associated with CLuster IP of OEM APP can be resolved both way
    
    2. Install on a shared disk/volume which holds the binaries and the gc_inst directory.
    >>> Install OEM APP on shared disk [keeping all the ORACLE_HOMEs on shared storage associated in cluster]

    3. The Inventory location must failover to the surviving node.
    >>> keeping Inventory on shared storage associated in cluster

    4. The software owner and time zone parameters must be the same on all cluster member nodes that will host this Oracle Management Service (OMS)
    >>> create users with same userID and username. MAker sure to configure NTP and same correct timezone 

Setting Up Shared Inventory
---------------------------
Use the following steps to set up shared inventory:
    1. Create your new ORACLE_BASE directory on shared storage:
      $ mkdir /oemapp/data01/orabase/
      $ chown -R oracle:oinstall /oemapp/data01/orabase/

    2. Create the Oracle Inventory directory under the new ORACLE_BASE:
      $ mkdir /oemapp/data01/orabase/oraInventory

    3. Create the oraInst.loc file in shared location
      $ vi /oemapp/data01/orabase/oraInst.loc
      ##Enter the path information to the Oracle Inventory directory and specify the group of the software owner as the oinstall user
      inventory_loc=/oemapp/data01/orabase/oraInventory
      inst_group=oinstall

Installing the Software
-----------------------
Refer to the following steps when installing the software:
   
    1. Point to the inventory location file oraInst.loc (under the ORACLE_BASE in this case), as well as specifying the host name of the virtual group(DNS form oem apps in this case)
       and install the oracle binaries on shared storage:
      $ ./runInstaller -invPtrloc /oemapp/data01/orabase/oraInst.loc ORACLE_HOSTNAME=oemapp.robi.com.bd -debug

when for putting middleware home location put the below:

/oemapp/data01/OEMAPP/middleware

user:weblogic(oem4Robi)
nodemanager(oem4Robi)

drop the repository from oemdb by following command:
emca -deconfig dbcontrol db -repos drop -SYS_PWD oem4Robi -SYSMAN_PWD oem4Robi -HOST oemdb.robi.com.bd

mkdir -p /oemdb/data01/OEMDB/data/mgmt/

run the root script on current node first and switch over and then run from node 2:

/oemapp/data01/OEMAPP/middleware/oms/allroot.sh

    2. Install Oracle Management Services on cluster member Host1(oemapp in our case).

    3. Continue the remainder of the installation normally.

    4. Once completed, copy the files oraInst.loc and oratab to /etc on all cluster member hosts (Host2, Host3, ...)


Starting Up Services
--------------------
Ensure that you start your services in the proper order. Use the order listed below:

    1. Establish the IP address on the active node.

    2. Start the TNS listener (if it is part of the same failover group).

    3. Start the database (if it is part of the same failover group).

    4. Start Cloud Control using emctl start oms

    5. Test functionality


Configuring the Software Library
--------------------------------

The software library location must be accessed by all Management Services. The configuration of software library is not performed during installation and needs to be configured post-install using the Enterprise Manager Console:

    1. On the Enterprise Manager home page, from the Setup menu, select Provisioning and Patching, and then select Software Library.

    2. Click the Provisioning subtab.

    3. On the Provisioning page, click the Administration subtab.

    4. In the Software Library Configuration section, click Add to set the Software Library Directory Location to a shared storage that can be accessed by any host running the Management Service.
    
/oemapp/data01/OEMAPP/middleware/oms/bin/emctl start oms

This information is also available at: 

/oemapp/data01/OEMAPP/middleware/oms/install/setupinfo.txt

See below for information pertaining to your Enterprise Manager installation:


Use the following URL to access:
--------------------------------
1. Enterprise Manager Cloud Control URL: https://oemapp.robi.com.bd:7799/em
2. Admin Server URL: https://oemapp.robi.com.bd:7101/console

The following details need to be provided during the additional OMS install:

1. Admin Server Hostname: oemapp.robi.com.bd
2. Admin Server Port: 7101

 NOTE:
 An encryption key has been generated to encrypt sensitive data in the Management Repository. If this key is lost, all encrypted data in the Repository becomes unusable. Please run following command to backup the Management Service configuration including the emkey and keep the backup on another system:
 emctl exportconfig oms -dir  


http://docs.oracle.com/cd/E24628_01/doc.121/e24473/ha_setup.htm#CEGCCCDE

DBFS on Exadata, Normal Oracle Database and on Client


I prepared below action plan while installing DBFS on both exadata and on an external ETL machine to mount DBFS from exadata.


-- ---------------------------
-- serverside [if not exadata]
-- ---------------------------
1. Check to see if the "kernel-devel" package is installed.
#rpm -q kernel-devel
kernel-devel-2.6.18-128.el5
#If not, then install it

2. Determine the kernel directory.
# echo /usr/src/kernels/`uname -r`-`uname -p`
/usr/src/kernels/2.6.18-128.el5-x86_64

3. Install the FUSE package as the "root" user using the following commands, substituting your kernel 

directory.
--There may be WARNING regarding "--with-kernel",  just ignore it
--download fuse-2.7.3.tar.gz from "http://fuse.sourceforge.net/"
# tar -xzvf fuse-2.7.3.tar.gz
# cd fuse-2.7.3
# ./configure --prefix=/usr --with-kernel=/usr/src/kernels/2.6.18-128.el5-x86_64
# make
# make install
# /sbin/depmod
# /sbin/modprobe fuse
# chmod 666 /dev/fuse
# echo "/sbin/modprobe fuse" >> /etc/rc.modules
# chmod 744 /etc/rc.modules

4. install below rpms
fuse-libs-2.7.4-8.el5
fuse-devel-2.7.4-8.el5
fuse-2.7.4-8.el5

5. create DBFS tablespace
SQL> create bigfile tablespace tbs_dbfs datafile '+DATA2' size 200G autoextend off;

6. create DBFS user
SQL> create user dbfs_user identified by dbfs_passwd default tablespace tbs_dbfs quota unlimited on tbs_dbfs;
SQL> grant create session, create table, create procedure, dbfs_role to dbfs_user;

7. create the file system in tablespace 
cd $ORACLE_HOME/rdbms/admin
sqlplus dbfs_user/dbfs_passwd@DB

SQL> @dbfs_create_filesystem.sql tbs_dbfs my_dbfs  
--tbs_dbfs= DBFS TABLESPACE, my_dbfs=USER GIVEN NAME OF FILESYSTEM
[
-- for advanced setting use below command 
SQL> start dbfs_create_filesystem_advanced tbs_dbfs DBFS nocompress nodeduplicate noencrypt non-partition
]

[
--to drop a file system, this can be done using the "dbfs_drop_filesystem.sql" script with the file system 

name.
cd $ORACLE_HOME/rdbms/admin
sqlplus test/test

SQL> @dbfs_drop_filesystem.sql my_dbfs
--my_dbfs=USER GIVEN NAME OF FILESYSTEM which created in previous step
]

8. add oracle to fuse group
# groupadd fuse
# usermod -a -G fuse oracle

9. create dbfs mount point 
# mkdir /data01/DBFS_DIR
# chown oracle:oinstall /data01/DBFS_DIR

10. to let "dbfs os user(oracle)" provide access to other and root
# echo user_allow_other > /etc/fuse.conf -- to let "dbfs os user(oracle)" provide access to other and root 

11. using oracle user collect group id of oinstall group in the DB server 
$cat /etc/group|grep oinstall|cut -d: -f3 

12. using oracle user create a password file
$ echo dbfs_passwd > $HOME/passwd.txt
chmod 600 $HOME/passwd.txt #only oracle 

13. mount DBFS for only mounter
export ORACLE_HOME=;export TNS_ADMIN=;export 

LD_LIBRARY_PATH=$ORACLE_HOME/lib;$ORACLE_HOME/bin/dbfs_client dbfs_user@DEVDW  -o direct_io /data01/DBFS_DIR < 

$HOME/passwd.txt &
--or
-- for root also
export ORACLE_HOME=;export TNS_ADMIN=;export 

LD_LIBRARY_PATH=$ORACLE_HOME/lib;$ORACLE_HOME/bin/dbfs_client dbfs_user@DEVDW -o allow_root  -o direct_io 

/data01/DBFS_DIR < $HOME/passwd.txt &

14. unmount
fusermount -u /data01/DBFS_DIR

-- -------------------
-- IF DB is on Exadata 
-- -------------------

1. OS level COnfig
# usermod -a -G fuse oracle
# mkdir /u01/app/DBFS_DIR
# chown oracle:oinstall /u01/app/DBFS_DIR
# echo user_allow_other > /etc/fuse.conf

2. Create tablespace and user
create bigfile tablespace tbs_dbfs datafile '+DATA2' size 200G autoextend off;
create user dbfs_user identified by dbfs_passwd default tablespace tbs_dbfs quota unlimited on tbs_dbfs;
grant create session, create table, create procedure, dbfs_role to dbfs_user;
start dbfs_create_filesystem_advanced tbs_dbfs DBFS nocompress nodeduplicate noencrypt non-partition

3. create a file with password
$echo dbfs_passwd > $ORACLE_HOME/passwd.txt
#chown 700 /u01/app/oracle/product/11.2.0/dbhome_1/passwd.txt

4. Create mount script
export 

ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1;TNS_ADMIN=/u01/app/oracle/product/11.2.0/dbhome_1/network/

admin;export LD_LIBRARY_PATH=/u01/app/oracle/product/11.2.0/dbhome_1/lib;nohup 

/u01/app/oracle/product/11.2.0/dbhome_1/bin/dbfs_client dbfs_user@DEVDW -o allow_other,direct_io 

/u01/app/DBFS_DIR < /u01/app/oracle/product/11.2.0/dbhome_1/passwd.txt &
-- for root also
export 

ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1;TNS_ADMIN=/u01/app/oracle/product/11.2.0/dbhome_1/network/

admin;export LD_LIBRARY_PATH=/u01/app/oracle/product/11.2.0/dbhome_1/lib;
nohup /u01/app/oracle/product/11.2.0/dbhome_1/bin/dbfs_client dbfs_user@DEVDW -o allow_root,direct_io 

/u01/app/DBFS_DIR < /u01/app/oracle/product/11.2.0/dbhome_1/passwd.txt &

5. command for unlount
fusermount -u /u01/app/DBFS_DIR


-- -----------
-- client side
-- -----------


1. Check to see if the "kernel-devel" package is installed.
#rpm -q kernel-devel
kernel-devel-2.6.18-128.el5
#If not, then install it

2. Determine the kernel directory.
# echo /usr/src/kernels/`uname -r`-`uname -p`
/usr/src/kernels/2.6.18-128.el5-x86_64

3. Install the FUSE package as the "root" user using the following commands, substituting your kernel 

directory.
--There may be WARNING regarding "--with-kernel",  just ignore it
# tar -xzvf fuse-2.7.3.tar.gz
# cd fuse-2.7.3
# ./configure --prefix=/usr --with-kernel=/usr/src/kernels/2.6.18-128.el5-x86_64
# make
# make install
# /sbin/depmod
# /sbin/modprobe fuse
# chmod 666 /dev/fuse
# echo "/sbin/modprobe fuse" >> /etc/rc.modules
# chmod 744 /etc/rc.modules

4. install below rpms
fuse-libs-2.7.4-8.el5
fuse-devel-2.7.4-8.el5
fuse-2.7.4-8.el5

5. permit other user to execute dbfs_client
#chmod 755 <$ORACLE_HOME>/bin/dbfs_client

6. permit other user to read tnsnames.ora
#chmod 755 <$ORACLE_HOME>/network/admin/tnsnames.ora

7. create OS dbfsuser and group
#groupadd -g 101 fuse [with same ID as server]
#create user dbfs_user -g fuse

8. create DBFS mount point
#mkdir /data01/DBFS_DIR
#chown oracle:fuse /data01/DBFS_DIR
#chmod 775 /data01/DBFS_DIR

9. create password file for dbfs  
#echo dbfs_passwd > $HOME/passwd.txt
#chown oracle:fuse $HOME/passwd.txt
#chmod 640 $HOME/passwd.txt

10. mount DBFS for only mounter
export ORACLE_HOME=;export TNS_ADMIN=;export 

LD_LIBRARY_PATH=$ORACLE_HOME/lib;$ORACLE_HOME/bin/dbfs_client dbfs_user@DEVDW  -o direct_io /data01/DBFS_DIR < 

$HOME/passwd.txt &
--or
-- for root also
export ORACLE_HOME=;export TNS_ADMIN=;export 

LD_LIBRARY_PATH=$ORACLE_HOME/lib;$ORACLE_HOME/bin/dbfs_client dbfs_user@DEVDW -o allow_root  -o direct_io 

/data01/DBFS_DIR < $HOME/passwd.txt &

11. with DBFS mounted create directory fully accessible by both client user & server group
#mkdir /data01/DBFS_DIR//EXT_DIR/
#c€hown dbfs_user: /data01/DBFS_DIR/EXT_DIR  # here 

owner=local user, group = DBFS DB oinstall group

12. unmount
fusermount -u /data01/DBFS_DIR