Pre-Installation tasks of Installing Oracle 11g Real Application Cluster (RAC) (11.1.0.6) 32-bit on  CentOS  4 Update 5 X86 By Bhavin Hingu

To create 2-node RAC using NAS (iSCSI base nfs), one would require 3 machines with the following hardware installed on it.

Per Node:

1 GB RAM, at least 8 GB of harddrive, 1 GHz CPU,
3 Network Interfaces: 1 for Public, 1 for Private, Cluster interconnect and 1 for storage network
    

Per Cluster:

1 Storage Server hosting Storage Operating System
1 Network HUB  (Public)
1 Network HUB  (Private -- cluster interconnet)
1 Network HUB  (Private -- Storage Server)
2 network cables for Public network (eth0)
2 network cables for private network for internode communication (eth1)
3 network cables for private network for storage Network (eth2)

I used the below hardware to build 2-node RAC cluster for this excercise.

  Oracle 11g R1 on CentOS EL 4 Update 5 requires below extra packages to be installed with the version same or higher than listed below.

Please execute the below command as root to make sure that you have this rpms installed. If not installed, then download and install

          using YUM.

Kernel Parameters:

Oracle recommends that you set shared memory segment attributes as well as semaphores to the following values.
If not set, database instance creation will fail. I added the following lines to /etc/sysctl.conf file. Every OS process needs
semaphore where It waits on for the resources. For more on semaphore, please read the UNIX os documents.
Oracle 10g RAC requires to have 1GB of RAM available on each node to sucessfully install 11g RAC. Well, I have
managed to install it with 512 MB RAM. You will get warning during checking of pre-requise step of installation
step which you can ignore. Please goto Adding an Extra Swapspace if you want to have an extra swapspace added.

 NOTE: If the current value for any parameter is higher than the value listed in this table, then do not change
              the value of that parameter.

                   To see the current setting in the kernel, please use the below command.

/sbin/sysctl -a | grep sem      -- for semmsl, semmns, semopm, semmni
/sbin/sysctl -a | grep shm      -- for shmall, shmmax, shmmni
/sbin/sysctl -a | grep file-max
/sbin/sysctl -a | grep ip_local_port_range
/sbin/sysctl -a | grep rmem_default

Please add/change the appropriate variables value in the /etc/sysctl.conf file as shown below.

[root@node2-pub ~]# cat /etc/sysctl.conf
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and
# sysctl.conf(5) for more details.
 
# Controls IP packet forwarding
net.ipv4.ip_forward = 0
 
# Controls source route verification
net.ipv4.conf.default.rp_filter = 1
 
# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0
 
# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0
 
# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1
 
# Extra parameters For 11g RAC installation
 
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.shmall = 2097152
kernel.sem = 250 32000 100 128
fs.file-max = 6553600
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 4194304
net.core.wmem_default = 262144
net.core.wmem_max= 262144
net.core.rmem_max = 4194304
 
[root@node2-pub ~]# 

After adding these lines to /etc/sysctl.conf, please run the below command as root to make them enabled.

[root@node2-pub ~]#  sysctl -p

groupadd -g 900 dba

groupadd -g 901 oinstall
useradd -u 900 -g oinstall -G dba oracle
passwd oracle

   As per OFA, oracle base directory has the path: /mount_point/app/oracle_sw_owner where,
   mount_point is the mount point directory for the file system that will contain the Oracle software. I have used
                     /u01 for the mount point directory. However, you could choose another mount point directory,
                     such as /oracle or /opt/oracle.

[root@node2-pub ~]# mkdir -p /u01/app/crs
[root@node2-pub ~]# mkdir -p /u01/app/asm
[root@node2-pub ~]# mkdir -p /u01/app/oracle
[root@node2-pub ~]# mkdir -p /u02/ocfs2
[root@node2-pub root]# chown -R oracle:oinstall /u01
[root@node2-pub root]# chown -R oracle:oinstall /u02
[root@node2-pub root]# chmod -R 775 /u01/app/oracle
[root@node2-pub root]# chmod -R 775 /u01

  Every node in the cluster must have access to the shared disk. So the shared disk must support the concurrent access to all
  nodes in cluster in order to successfully build 11g RAC. There are so many Storage Management OSes are available in the market
  that allows you to build NAS/SAN appliances. I have chosen 'openfiler' as it is Linux 2.6 kernel based Storage Management OS and
  it is powered by open source applications such as iSCSI.  You can attach any type of external disks to this server and that can be
  presented as iscsi devices. You can then create volume groups on this device(s) and later these volume groups are discovered as
  new scsi devices on the RAC nodes.

Steps Involved to install/configure iSCSI based IP SAN.

(1) install openfiler OS

(2) Attach external disks to this server
  
     I have attached the 500 GB WD USB (MyBook) hard drive.
     I planned to create 4 volumes for ASM file system each of 100GB and one for OCFS2

     asm-dsk1, asm-dsk2, asm-dsk3, asm-dsk4 and ocfs-dsk
   
     ASM: DATA and FLASH volume for database files and Flash Recovery Area files.
     OCFS2: OCR, Voting Disks.

(3) Configure openfiler setup -- iscsi-target/volume groups/shared access etc.

I have followed the above guide to configure the openfiler system and create the volumes for ASM disks and OCFS disks.
Below are the sample screen shots for my openfiler setup.

The external disks is presented to the server as SCSI disks as shown below. In my case it is /dev/sda. I have created physical volume on
this device and then created volume group rac11g on entire physical volume.



The below 5 volumes  are created under the rac11g volume group.



Also make sure that each volumes allows the shared access to all the nodes in the cluster. You can do that by clicking the 'Edit' link on the above screen
for each volume name as shown below.



The below screen shows that both the nodes in the cluster has shared access on the storage. Click on the General tab to add/modify the rac nodes info.
The network information provided in this table are provate network for the shared storage.



At the End, make sure that iSCSI protocol is enabled in openfiler. You can enable by clicking on the services tab.




(4) Discovering the volumes to the RAC nodes as scsi devices

NOTE: make sure that SELinux and firewall has been disabled on all the RAC nodes. If not, then disable by "Administraion Sections"
--> "system settings" --> "security Level"

Make Sure that the service iscsi-target is running on nas-server:
 
[root@nas-server ~]# service iscsi-target status
ietd (pid 10688) is running...
[root@nas-server ~]#
 
on all the RAC nodes:
 
Make sure that the iscsi-initiator is installed on RAC nodes. If not, then downlaod the specified RPM
and istall them.
 
[root@node1-pub rpms]# rpm -qa | grep iscsi
iscsi-initiator-utils-4.0.3.0-5
 
[root@node1-pub rpms]# service iscsi start -- if not already running
 
[root@node1-pub rpms]# service iscsi status
iscsid (pid 3256 3253) is running...
 
Configure nodes for the Disk Discovery:
 
Edit the /etc/iscsi.conf and assign the address of the iscsi target
to the below parameter:
 
 
# --------------------------
# Discovery Address Category
# --------------------------
# To localize parameters to targets found on a particular discovery address
# (i.e., to override the global settings), you need to use the
# "DiscoveryAddress" setting. The format for the "DiscoveryAddress" setting is
# a.b.c.d, a.b.c.d:e (where e is a TCP port number), or an instance name. An
# example of these settings would be:
#
#DiscoveryAddress=10.4.100.0
#  or
#DiscoveryAddress=10.4.100.1:3260
#  or
DiscoveryAddress=192.168.1.33 # <-- This is the address of the nas-server
#
 
(b)
 
Reboot All the Nodes and run the iscsi-ls command to see if the volumes have been discovered on the RAC nodes as scsi devices.
 
(c)
 
[root@node2-pub rpms]# iscsi-ls
*******************************************************************************
SFNet iSCSI Driver Version ...4:0.1.11-4(15-Jan-2007)
*******************************************************************************
TARGET NAME             : iqn.2006-01.com.openfiler:rac11g.ocfs-dsk
TARGET ALIAS            :
HOST ID                 : 0
BUS ID                  : 0
TARGET ID               : 0
TARGET ADDRESS          : 192.168.1.33:3260,1
SESSION STATUS          : ESTABLISHED AT Sun Sep  2 18:49:57 CDT 2007
SESSION ID              : ISID 00023d000001 TSIH 1200
*******************************************************************************
TARGET NAME             : iqn.2006-01.com.openfiler:rac11g.asm-dsk4
TARGET ALIAS            :
HOST ID                 : 1
BUS ID                  : 0
TARGET ID               : 0
TARGET ADDRESS          : 192.168.1.33:3260,1
SESSION STATUS          : ESTABLISHED AT Sun Sep  2 18:49:57 CDT 2007
SESSION ID              : ISID 00023d000001 TSIH 1000
*******************************************************************************
TARGET NAME             : iqn.2006-01.com.openfiler:rac11g.asm-dsk3
TARGET ALIAS            :
HOST ID                 : 2
BUS ID                  : 0
TARGET ID               : 0
TARGET ADDRESS          : 192.168.1.33:3260,1
SESSION STATUS          : ESTABLISHED AT Sun Sep  2 18:49:57 CDT 2007
SESSION ID              : ISID 00023d000001 TSIH 1100
*******************************************************************************
TARGET NAME             : iqn.2006-01.com.openfiler:rac11g.asm-dsk2
TARGET ALIAS            :
HOST ID                 : 3
BUS ID                  : 0
TARGET ID               : 0
TARGET ADDRESS          : 192.168.1.33:3260,1
SESSION STATUS          : ESTABLISHED AT Sun Sep  2 18:49:57 CDT 2007
SESSION ID              : ISID 00023d000001 TSIH 1300
*******************************************************************************
TARGET NAME             : iqn.2006-01.com.openfiler:rac11g.asm-dsk1
TARGET ALIAS            :
HOST ID                 : 4
BUS ID                  : 0
TARGET ID               : 0
TARGET ADDRESS          : 192.168.1.33:3260,1
SESSION STATUS          : ESTABLISHED AT Sun Sep  2 18:49:58 CDT 2007
SESSION ID              : ISID 00023d000001 TSIH 1400
*******************************************************************************
[root@node2-pub rpms]#
 
 
(4) Map the Volumes on the iscsi-target (nas-server) to the Disks disceverd on the local RAC nodes.
 
    Host ID       Target ID                                       discovered as
   
      0           iqn.2006-01.com.openfiler:rac11g.ocfs-dsk          ---
      1           iqn.2006-01.com.openfiler:rac11g.asm-dsk4          ---
      2           iqn.2006-01.com.openfiler:rac11g.asm-dsk3          ---
      3           iqn.2006-01.com.openfiler:rac11g.asm-dsk2          ---
      4           iqn.2006-01.com.openfiler:rac11g.asm-dsk1          ---
 
Now, run the below command to find out the "Attached" devices to the Host IDs. The scsi Id in this
output maps to the Host ID on the "iscsi-ls" output.
 
[root@node2-pub rpms]# dmesg | grep Attached
 
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
Attached scsi disk sdb at scsi1, channel 0, id 0, lun 0
Attached scsi disk sdc at scsi2, channel 0, id 0, lun 0
Attached scsi disk sdd at scsi3, channel 0, id 0, lun 0
Attached scsi disk sde at scsi4, channel 0, id 0, lun 0
 
In first line, scsi0 (Host Id 0) has device "sda" attahced to it. So, By filling the above table
with this information gives the mapping of discovered Disks at client to its actual Volums on the iscsi-target.
 
 Host ID          Target ID                                 Volume Name discovered as
   
      0           iqn.2006-01.com.openfiler:rac11g.ocfs-dsk   ocfs-dsk           sda
      1           iqn.2006-01.com.openfiler:rac11g.asm-dsk4   asm-dsk4           sdb
      2           iqn.2006-01.com.openfiler:rac11g.asm-dsk3   asm-dks3           sdc
      3           iqn.2006-01.com.openfiler:rac11g.asm-dsk2   asm-dsk2           sdd
      4           iqn.2006-01.com.openfiler:rac11g.asm-dsk1   asm-dsk1           sde
 
 

Download and Install the required rpms:

Please download the below rpms from Oracle's website and install them as shown.

ocfs2-2.6.9-55.0.2.EL-1.2.5-6.i686.rpm
ocfs2-tools-1.2.4-1.i386.rpm
ocfs2-tools-debuginfo-1.2.4-1.i386.rpm
ocfs2-tools-devel-1.2.4-1.i386.rpm

[root@node1-pub rpms]# rpm -ivh ocfs2-2.6.9-55.0.2.EL-1.2.5-6.i686.rpm \
> ocfs2-tools-1.2.4-1.i386.rpm \
> ocfs2console-1.2.4-1.i386.rpm \
> ocfs2-tools-debuginfo-1.2.4-1.i386.rpm \
> ocfs2-tools-devel-1.2.4-1.i386.rpm
Preparing...                ########################################### [100%]
   1:ocfs2-tools            ########################################### [ 20%]
   2:ocfs2-2.6.9-55.0.2.EL  ########################################### [ 40%]
   3:ocfs2console           ########################################### [ 60%]
   4:ocfs2-tools-debuginfo  ########################################### [ 80%]
   5:ocfs2-tools-devel      ########################################### [100%]
[root@node1-pub rpms]#


Configure OCFS2:

The configuration of the ocfs2 resides in the /etc/ocfs2/cluster.conf file that is generated by running ocfs2console command and then is prapogated
to different nodes in the cluster. Below are the screen shots for the ocfs2 configuration process.

Running ocfs2console command line utility brings this screen.



You can close this box as you can enable the o2cb service later.



Click Cluster  --> Configure Nodes. Add node names for each node one-by-one.
Make sure to add exact same node name as it has been returned by the `hostname` command.



My host name names are node1-pub.hingu.net and node2-pub.hingu.net for node 1 and node 2 respectively.




Prapogate the files to all the nodes in cluster:



So, the above settings of "Name" (node1-nas and node2-nas) in the ocfs2 configuraion will lead to
below error when I try to enable o2cb service.

[root@node1-pub rpms]# /etc/init.d/o2cb enable
Writing O2CB configuration: OK
Starting O2CB cluster ocfs2: Failed
Cluster ocfs2 created
Node node1-nas added
Node node2-nas added
o2cb_ctl: Configuration error discovered while populating cluster ocfs2.  None of its nodes were considered local.  A node is considered local when its node name in the configuration matches this machine's host name.
Stopping O2CB cluster ocfs2: OK
[root@node1-pub rpms]#
 
So, stop o2cb service, open the /etc/ocfs2/cluster.conf file and update the hostname value to the one that is returned by `hostname` command.
Do not update  the IP. Start the service and load it again and the error should go away.
--=================

[oracle@node2-pub ~]$ cat /etc/ocfs2/cluster.conf
node:
        ip_port = 7777
        ip_address = 192.168.0.11
        number = 0
        name = node1-pub.hingu.net
        cluster = ocfs2
 
node:
        ip_port = 7777
        ip_address = 192.168.0.22
        number = 1
        name = node2-pub.hingu.net
        cluster = ocfs2
 
cluster:
        node_count = 2
        name = ocfs2
--================

[root@node2-pub rpms]# /etc/init.d/o2cb load
Loading module "configfs": OK
Creating directory '/config': OK
Mounting configfs filesystem at /config: OK
Loading module "ocfs2_nodemanager": OK
Loading module "ocfs2_dlm": OK
Loading module "ocfs2_dlmfs": OK
Creating directory '/dlm': OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
[root@node2-pub rpms]# /etc/init.d/o2cb status
Module "configfs": Loaded
Filesystem "configfs": Mounted
Module "ocfs2_nodemanager": Loaded
Module "ocfs2_dlm": Loaded
Module "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Offline
[root@node2-pub rpms]#
 
Configure o2cb to startup at Boot time:
 
[root@node2-pub rpms]# /etc/init.d/o2cb configure
Configuring the O2CB driver.
 
This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on
boot.  The current values will be shown in brackets ('[]').  Hitting
<ENTER> without typing an answer will keep that current value.  Ctrl-C
will abort.
 
Load O2CB driver on boot (y/n) [n]: y
Cluster to start on boot (Enter "none" to clear) [ocfs2]:
Specify heartbeat dead threshold (>=7) [7]:
Specify network idle timeout in ms (>=5000) [10000]:
Specify network keepalive delay in ms (>=1000) [5000]:
Specify network reconnect delay in ms (>=2000) [2000]:
Writing O2CB configuration: OK
Starting O2CB cluster ocfs2: OK
[root@node2-pub rpms]#
 
 
[root@node2-pub rpms]# chkconfig --add ocfs2
[root@node2-pub rpms]# chkconfig --add o2cb
[root@node2-pub rpms]# mkdir -p /u02/ocfs2 -- ocfs2 mountpoint

Format the filesystem

Select the device that you want to format. In my case it is /dev/sda1 that I want to format for ocfs2.








I got the below error while ocfs2 being mounted on the another node.

ERROR: While mounting the same filesystem on another node.
 
[root@node2-pub ~]# mount -t ocfs2 /dev/sda1 /u02/ocfs2
mount.ocfs2: Transport endpoint is not connected while mounting /dev/sda1 on /u02/ocfs2. Check 'dmesg' for more information on this error.
 
Solution: Make sure that SELinux and Firewall has been disabled. If not, Disable them and reboot all the nodes,
and it should be ok.

On the other node, you only need to execute the below command to show these disks up there.

[root@node2-pub ~]# /etc/init.d/oracleasm scandisks
[root@node2-pub ~]# /etc/init.d/oracleasm listdisks
DSK1
DSK2
DSK3
DSK4

Oracle Documents