Pre-Installation tasks of Installing Oracle 9iR2 Real Application Cluster (RAC) (9.2.0.1) 32-bit on  RedHat AS 3 x86 (RHEL3) / CentOS 3 x86 By Bhavin Hingu

To create 2-node RAC, one would requre 2 machine with the following hardware installed on it.

Per Node:

512MB RAM, at least 6 GB of harddrive, 1 GHz CPU,
1 Firewire Controller,  1 Firewire Cable
2 NIC ethernet card (one for public and another for private / interconnect network)

Per Cluster:

1 Shared Harddrive
1 firewire HUB + 1 firewire cable (for cluster with more than 2 node)
1 Network HUB + 1 network cable (for cluster with more than 2 node)
1 crossover network cable (for cluster with 2 node)
n number of  network cable for private network for internode communication (for cluster with n nodes where n >=3)
n number of network cable for public network (for cluster with n nodes where n >=3)

I used the below hardware to build my 2-node rac.

Goto otn.oracle.com and download the appropriate Oracle 9iR2 Software into the /tmp. Make Sure You have enough
space under this mount point. You can check this using df command. I downloaded following 3 files from OTN for my 32-bit Linux box
under the /tmp directory. Oracle supplies the Clusterware software as part of the Regular 9i RDBMS software by default for Linux.
So you do not need to ownload it like we do in case of 10g.

  After you install the Linux system and before you start installing Oracle10g software, please make sure that you have the below
  packages installed on your   Linux box, else you will get error(s) during the installation process.

Please execute the below comand as root to make sure that you have this rpms installed. If not installed, then download them

from appropriate linux site.

rpm -q make gcc glibc compat-db compat-gcc compat-gcc-c++ compat-libstdc++ \
       compat-libstdc++-devel openmotif21 setarch libaio libaio-devel pdksh ncurses4

Oracle 9i RAC requires to have 512 of RAM available on each node to sucessfully install 9i RAC. Well, You will get warning
during checking of pre-requise step of installation step if you do not have required memorywhich you can ignore. Please
goto Adding an Extra Swapspace if you want to have an extra swapspace added.

Kernel Parameters:

Please goto Setting Up kernel Parameter to set the kernel parameters.

  You need OS “oracle” user account created which owns the Oracle software. Oracle Software installation
  needs to be proceeds by this account. Oracle software installation (without Companion CD) requires 6 GB
  of free space available for the ORACLE_BASE directory. Please make sure that the mount point where you

groupadd -g 900 dba

groupadd -g 901 oinstall
useradd -u 900 -g oinstall -G dba oracle
passwd oracle

   As per OFA, oracle base directory has the path: /mount_point/app/oracle_sw_owner where,
   mount_point is the mount point directory for the file system that will contain the Oracle software. I have used
                     /u01 for the mount point directory. However, you could choose another mount point directory,
                     such as /oracle or /opt/oracle.

[root@node1-pub root]# mkdir -p /u01/app/oracle  -- ORACLE_BASE
[root@node1-pub root]# mkdir -p /u01/app/oracle/product/9.2.0  -- ORACLE_HOME
[root@node1-pub root]# mkdir  -p /u01/app/oracle/product/9.2.0
[root@node1-pub root]# mkdir  -p /u01/app/oracle/product/9.2.0/oracm/log
[root@node1-pub root]# mkdir  -p /u01/app/oracle/product/9.2.0/network/log
[root@node1-pub root]# mkdir  -p /u01/app/oracle/product/9.2.0/network/trace
[root@node1-pub root]# mkdir  -p /u01/app/oracle/product/9.2.0/rdbms/log
[root@node1-pub root]# mkdir  -p /u01/app/oracle/product/9.2.0/rdbms/audit
[root@node1-pub root]# mkdir  -p /u01/app/oracle/product/9.2.0/network/agent/log
[root@node1-pub root]# mkdir  -p /u01/app/oracle/product/9.2.0/network/agent/reco
[root@node1-pub root]# mkdir -p /u02/oradata/ocfs  -- For database to be stored on ocfs
[root@node1-pub root]# mkdir -p /u02/oradata/raw   -- For database to be stored on raw devices (volumes)
[root@node1-pub root]# chown -R oracle:oinstall /u01
[root@node1-pub root]# chown -R oracle:oinstall /u02
[root@node1-pub root]# chmod -R 775 /u01/app/oracle
[root@node1-pub root]# chmod -R 775 /u02

(1)  Create the /etc/rhosts.equiv file and add all the node name in the clusters (public as well as private) along with oracle as os user.
         This will allow only oracle to perform repote operations on the nodes withoug being asked for the password.

	[root@node1-pub root]# cat > /etc/hosts.equiv
	+node1-pub oracle
	+node1-prv oracle
	+node2-pub oracle
	+node2-prv oracle
 
	[root@node1-pub root]# cat  /etc/hosts.equiv
	+node1-pub oracle
	+node1-prv oracle
	+node2-pub oracle
	+node2-prv oracle
	[root@node1-pub root]# chmod 600 /etc/hosts.equiv
	[root@node1-pub root]# ls -l /etc/hosts.equiv
	-rw-------    1 root     root           72 Dec 14 22:21 /etc/hosts.equiv
	[root@node1-pub root]#

[root@node1-pub root]# rpm -qa rsh*
rsh-0.17-17.6
rsh-server-0.17-17.6

[root@node1-pub xinetd.d]# service xinetd restart
Stopping xinetd:                                           [  OK  ]
Starting xinetd:                                           [  OK  ]
[root@node1-pub xinetd.d]# chkconfig rsh on
[root@node1-pub xinetd.d]# chkconfig rlogin on
 rsh node1-prv date

rsh node1-prv date
rsh node2-prv date
rsh node1-pub date
rsh node2-pub date

rlogin node1-prv
rlogin node2-prv
rlogin node1-pub
rlogin node2-pub

Create the following symbolic links to put gcc296 and g++296 in the $PATH variable. These are required during the relink process of Oracle Software binaries.

	mv /usr/bin/gcc /usr/bin/gcc323
	mv /usr/bin/g++ /usr/bin/g++323
	ln -s /usr/bin/gcc296 /usr/bin/gcc
	ln -s /usr/bin/g++296 /usr/bin/g++  

Executing hostname should return the fully qualified name (hostname.domainname).

Setup oracle User Environment:
 
Add the below parameters into the .bash_profile file of oracle user. Usually this file is located under /home/oracle.

	export ORACLE_BASE=/u01/app/oracle
	export ORACLE_HOME=$ORACLE_BASE/product/9.2.0
	export ORACLE_SID=RacDB1      # on node2, it is RacDB2
	export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:/usr/local/lib
        export LD_ASSUME_KERNEL=2.4.19
	export TNS_ADMIN=$ORACLE_HOME/network/admin
	export ORA_NLS33=$ORACLE_HOME/ocommon/nls/admin/data
	export ORACLE_TERM=xterm
	export PATH=$PATH:$ORACLE_HOME/bin:/opt/bin:/bin:/usr/bin:/usr/local/bin:/usr/sbin:/usr/X11R6/bin:/usr/local/java/bin:.
	export CLASSPATH=$ORACLE_HOME/jdbc/lib/classes12.zip:$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib:$ORACLE_HOME/network/jlib:.

Obtaining Extra 9i patchsets:

Along with the base software (9.2.0.1) you also need to download and apply patchsets on top of the base release to fix most of the bugs
that comes with base release (9.2.0.1). Please download the below patches from the http://metalink.oracle.com. If you are installing for
experimental / learning purpose then you do not need to apply all the patches.

p3006854_9204_LINUX.zip:  This patch is used to fix the below error generated by executing runInstaller.

Error occurred during initialization of VM
Unable to load native library: /tmp/OraInstall2005-12-16_02-19-25AM/jre/lib/i386/libjava.so: symbol __libc_wait, version GLIBC_2.0 not defined in file libc.so.6 with link time reference

This patch builds /etc/libwait.so and ld.so.preload files. The second file is used to load the module
libcwait.so on every boot of OS.

p2617419_10102_GENERIC.zip: This patch supply the opatch utility which is used to apply oracle patch.
p3095277_9204_LINUX.zip: This patch is required to upgrade the 9.2.0.1 database server to 9.2.0.4 server.
p3119415_9204_LINUX.zip: This patch fixes the linking error during the installation of intelligent agent file: ins_oemagent.mk

  Every node in the cluster must have access to the shared disk. So the shared disk must support the concurrent access to all
  nodes in cluster in order to successfully build 10g RAC. I chose firewire Disk as a shared storage media because it is a cost
  effective solution ifyou just want to have hands-on practice on 10g RAC without investing more money. After you install the
  Redhat LINUX AS 3 system into both the node, please goto the http://oss.oracle.com/projects/firewire/files and  download the
  appropriate firewire kernel to support firewire HD. I downloaded and installed the below rpms.

 [root@localhost root]# uname -r
  2.4.21-37.EL

kernel-2.4.21-27.0.2.ELorafw1.i686.rpm

 [root@localhost root]# rpm -ivh --force kernel-2.4.21-27.0.2.ELorafw1.i686.rpm

 This will also update the /etc/grub.conf file with the added entry of this new firewire kernel. in the below file, default is set to 1
 which means that  the system will use the original kernel by default. If you want to make the newly added firewire kernel
 as default, you can simply change the default=1 to default=0. It is required to set this kernel to default in the situation if this
 node is restarted  by hangcheck-timer or for anyreason, then it should be rebooted with the right kernel.

 [root@node2-pub root]# cat /etc/grub.conf
 # grub.conf generated by anaconda
 #
 # Note that you do not have to rerun grub after making changes to this file
 # NOTICE:  You have a /boot partition.  This means that
 #          all kernel and initrd paths are relative to /boot/, eg.
 #          root (hd0,0)
 #          kernel /vmlinuz-version ro root=/dev/hda2
 #          initrd /initrd-version.img
 #boot=/dev/hda
 default=0
 timeout=10
 splashimage=(hd0,0)/grub/splash.xpm.gz
 title CentOS (2.4.21-27.0.2.ELorafw1)
        root (hd0,0)
        kernel /vmlinuz-2.4.21-27.0.2.ELorafw1 ro root=LABEL=/
        initrd /initrd-2.4.21-27.0.2.ELorafw1.img
 title CentOS-3 (2.4.21-37.EL)
        root (hd0,0)
        kernel /vmlinuz-2.4.21-37.EL ro root=LABEL=/
        initrd /initrd-2.4.21-37.EL.img

 Also update the /etc/modules.conf file and add the below lines at the end of file on BOTH THE NODES. This will load the firewire
 kernel modules and drivers at reboot.

Download and Install the required rpms:
Please download the below rpms from Oracle's website and install them as shown.

ocfs-2.4.21-EL-1.0.14-1.i686.rpm (For UniProcessor)
ocfs-2.4.21-EL-smp-1.0.14-1.i686.rpm (For SMPs)
ocfs-tools-1.0.10-1.i386.rpm
ocfs-support-1.1.5-1.i386.rpm


[root@node1-pub root]# rpm -Uvh /rpms/ocfs-2.4.21-EL-1.0.14-1.i686.rpm \
>                               /rpms/ocfs-tools-1.0.10-1.i386.rpm \
>                               /rpms/ocfs-support-1.1.5-1.i386.rpm
Preparing...                ########################################### [100%]
   1:ocfs-support           ########################################### [ 33%]
   2:ocfs-2.4.21-EL         ########################################### [ 67%]
Linking OCFS module into the module path [  OK  ]
   3:ocfs-tools             ########################################### [100%]
[root@node1-pub root]#
[root@node1-pub root]# cat /etc/ocfs.conf
#
# ocfs config
# Ensure this file exists in /etc
#
                                                                                                                            
        node_name = node1-prv
        ip_address = 192.168.203.1
        ip_port = 7000
        comm_voting = 1
        guid = 83C074F61764EA84570200065BBED79C
[root@node1-pub root]#

Loading OCFS Module:

[root@node1-pub root]# load_ocfs
/sbin/modprobe ocfs node_name=node2-pub ip_address=192.168.203.2 cs=1843 guid=238426EC6845F952C83A00065BAEAE7F ip_port=7000 comm_voting=1
modprobe: Can't locate module ocfs
load_ocfs: insmod failed

If you get the above error follow the below steps to fix this:
Verify that you have ocfs.o module under /lib/modules/2.4.21-27.0.2.ELorafw1/kernel/drivers/addon/ocfs/ directory.

[root@node1-pub root]# ls -l /lib/modules/2.4.21-27.0.2.ELorafw1/kernel/drivers/addon/ocfs/ocfs.o lrwxrwxrwx    1 root     root           38 Dec 19 23:14 /lib/modules/2.4.21-27.0.2.ELorafw1/kernel/drivers/addon/ocfs/ocfs.o -> /lib/modules/2.4.21-EL-ABI/ocfs/ocfs.o


IF THIS FILE EXISTS THEN:
     open the /sbin/load_ocfs file using vi or anyother editor and change the below line as shown.
     (Line Number 93)

# If you must hardcode an absolute module path for testing, do it HERE.
# MODULE=/path/to/test/module/ocfsX.o
Change to:
# If you must hardcode an absolute module path for testing, do it HERE.
 MODULE=/lib/modules/2.4.21-27.0.2.ELorafw1/kernel/drivers/addon/ocfs/ocfs.o

IF THIS FILE DOES NOT EXISTS THEN:
Create an sybolic link as shown below.

mkdir /lib/modules/2.4.21-27.0.2.ELorafw1/kernel/drivers/addon/ocfs/
ln -s /lib/modules/2.4.21-EL-ABI/ocfs/ocfs.o /lib/modules/2.4.21-27.0.2.ELorafw1/kernel/drivers/addon/ocfs/ocfs.o

Now try again to load the same module
[root@node1-pub root]# load_ocfs

If you get the error again then moidify the /sbin/load_ocfs file as shown in the above step after creating the
symbolic link

[root@node1-pub root]# load_ocfs
/sbin/insmod /lib/modules/2.4.21-27.0.2.ELorafw1/kernel/drivers/addon/ocfs/ocfs.o node_name=node2-prv ip_address=192.168.203.2 cs=1843 guid=238426EC6845F952C83A00065BAEAE7F ip_port=7000 comm_voting=1
Warning: kernel-module version mismatch
        /lib/modules/2.4.21-27.0.2.ELorafw1/kernel/drivers/addon/ocfs/ocfs.o was compiled for kernel version 2.4.21-27.EL
        while this kernel is version 2.4.21-27.0.2.ELorafw1
Warning: loading /lib/modules/2.4.21-27.0.2.ELorafw1/kernel/drivers/addon/ocfs/ocfs.o will taint the kernel: forced load
  See http://www.tux.org/lkml/#export-tainted for information about tainted modules
Module ocfs loaded, with warnings

You may get the above warning but this may be OK.
Verify that the ocfs module is loaded by executing the below command.

[root@node1-pub root]# lsmod | grep ocfs
ocfs                  299104   0  (unused)

Creating and Mounting OCFS (Oracle Cluster File System):

Create the filesystem using mkfs:
Execute the below series of command from any one node to format the /dev/sda1 partition with the ocfs.

[root@node1-pub root]# mkfs.ocfs -F -b 128 -L /u02/oradata/ocr -m /u02/oradata/ocr -u 900 -g 901 -p 0755 /dev/sda1
Cleared volume header sectors
Cleared node config sectors
Cleared publish sectors
Cleared vote sectors
Cleared bitmap sectors
Cleared data block
Wrote root directory and system files
Updated global bitmap
Wrote volume header

Where  -b= blocksize
             m= mountpoint
             u= UID of oracle user
             g=GID of oinstall group
             p=permission

Mounting OCFS (Oracle Cluster File System): (Do this on both the node)

[root@node1-pub root]# mount -t ocfs /dev/sda2 /u02/oradata/ocfs

Add the below line into the /etc/fstab file to mount the ocfs automatically on every reboot of system.

/dev/sda2     /u02/oradata/ocfs    ocfs   _netdev 0 0

[root@node2-pub root]# service ocfs start
Loading OCFS:                                              [  OK  ]
[root@node2-pub root]# chkconfig ocfs on

Create the filesystem using "ocfstool" command line utility:
Please  follow the GUI screenshots of  creating and mounting ocfs filesystem. run the ocfstool from command
as shown below:
(1) Perform this step on Both the Nodes.



Cleck on the "Tasks" Button --> Select "Generate Configs".

---> Select  interface = eth1
             port = 7000 and
             Node Name = node1-prv (For node2, it would be node2-prv)



Confirm changes by looking into the /etc/ocfs.conf file. The contents of this file
should be like this:



(2)
Now, Click on the "Tasks" button and select 'Format". You will see the screen like below.
Select the appropriate value and click OK button.
You need to perform this step only from one node.





(3)
Now the /dev/sda2 is formated with the ocfs. Now this is the time to mount this filesystem.
Please perform this step on both the nodes. Click on the "Mount" button. You should see the
/dev/sda2 is mounted under /u02/oradata/ocfs mountpoint. Also confirm that you see both the
nodes in the "Configured Nodes" section.



After mounting the filesystem on both the nodes, you should see both the nodes
under the "Configured Nodes" tab.




Add the below line into the /etc/fstab file on both the nodes to mount the ocfs automatically on every reboot.

/dev/sda2     /u02/oradata/ocfs    ocfs   _netdev 0 0

perform this step on both the nodes.

Checking the Configuration of the hangcheck-timer Module:

Before Installing Oracle Real Application Cluster, We need to verify that the hangcheck-timer module is loaded and configured     correctly. The hangcheck-timer module monitors the Linux kernel for extended operating system hangs that could affect the        reliability of a RAC nodeand cause a database corruption. If a hang occures than the module restarts the node in seconds. There are hangcheck_tick and hangcheck_margin parameters that governs the behaviou of the modules:

Oracle Documents