oui-10009

This error occured while adding a new node in the existing cluster. The following workaround is used as follows:

Node 1:

cd $ORACLE_HOME/oui/bin

./addNode.sh -silent "CLUSTER_NEW_NODES={new nodename}" -logLevel trace –debug

You may want to browse "Node add" to get more information.

Node add

 

 

PRKP-1037 : Failed to create cluster Database

Step 1:

[oracle@orcl-experts.info bin]$ ./crs_stat
NAME=ora.SAAP.db
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE

Step 2 : crs_unregister ora.SAAP.db

Step 3: srvctl add database -d SAAP -o /data01/app/oracle/product/10.2.0/db/

[oracle@sabharwal11 bin]$ srvctl add instance -d SAAP -i SAAP1 -n sabharwal11
[oracle@sabharwal11 bin]$ srvctl add instance -d SAAP -i SAAP2 -n sabharwal12
[oracle@sabharwal11 bin]$ srvctl add instance -d SAAP -i SAAP3 -n sabharwal13

[oracle@sabharwal11 bin]$ srvctl stop database -d SAAP
[oracle@sabharwal11 bin]$ srvctl start database -d SAAP
[oracle@sabharwal11 bin]$ ./crs_stat
NAME=ora.SAAP.SAAP1.inst
TYPE=application
TARGET=ONLINE
STATE=ONLINE on sabharwal11

NAME=ora.SAAP.SAAP2.inst
TYPE=application
TARGET=ONLINE
STATE=ONLINE on sabharwal12

NAME=ora.SAAP.SAAP3.inst
TYPE=application
TARGET=ONLINE
STATE=ONLINE on sabharwal13

[oracle@sabharwal11 bin]$ srvctl status database -d SAAP
Instance SAAP1 is running on node sabharwal11
Instance SAAP2 is running on node sabharwal12
Instance SAAP3 is running on node sabharwal13

CRS-0210: Could not find resource

 

[oracle@orcl-experts1.com admin]$ srvctl stop nodeapps -n orcl-experts1.com

CRS-0210: Could not find resource 'ora.orcl-experts1.com.LISTENER_orcl-experts1.com.lsnr'.

[oracle@orcl-experts1.com admin]$ srvctl start nodeapps -n orcl-experts1.com

CRS-0210: Could not find resource ora.orcl-experts1.com.LISTENER_orcl-experts1.com.lsnr.

[oracle@orcl-experts1.com bin]$ ./crs_register ora.orcl-experts1.com.LISTENER_orcl-experts1.com.lsnr

[oracle@orcl-experts1.com bin]$ srvctl start nodeapps -n orcl-experts1.com
[oracle@orcl-experts1.com bin]$ srvctl stop nodeapps -n orcl-experts1.com
[oracle@orcl-experts1.com bin]$ srvctl start nodeapps -n orcl-experts1.com

NO Errors

 

CRS ownership - CRS did not start after node reboot - Metalink 428195.1

I had to deal with a maintenance which required disabling the auto start of CRS during server reboot:

#/etc/init.d/initcrs disable

I shutdown the RAC 2 node cluster and verified that there is no oracle process running and hand over the servers to Sysadmin. After the hardware maintenance before the 2nd reboot the sysadmin asked me to re enable the CRS auto start. In both nodes I did:

#/etc/init.d/initcrs enable

Sysadmin rebooted both the nodes. After 10 minutes I checked the process details and found that CRS stack is not running and the instances are not up. We had a deadline of 1 hour to bring up all the instances. Under the short time and pressure I started digging out the issue of why the CRS stack was not up. I tried to bring it up manually but it did not work.

I found that after remounting the file systems the permission of /oracle01 mount point (voting file and OCR file location) is:

-rw-r--r-- 1 root root 10240000 Apr 3 21:41 voting.dbf

Sysadmin and I decided to make an immediate change:

chown -R root:install /oracle01

and

we rebooted the machine.

It worked and during the node reboots process all the CRS and Oracle processes came up.

The maintenance was based on the mounting the existing file system on the new LUN.

clsssInitNative: connect failed, rc 9,Received status 3 from CSS. Waiting for good status .., Failure at final check of Oracle CRS stack. 10

Issue: I had issue while bringing up CRS in the 2nd node.The root.sh executions failed with the following message:

Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.

Failure at final check of Oracle CRS stack.
10

crsd.log

2007-10-26 17:33:37.510: [ CSSCLNT][3086960320]clsssInitNative: connect failed, rc 9

2007-10-26 17:33:37.511: [ CRSRTI][3086960320]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..

2007-10-26 17:33:38.545: [ COMMCRS][116607920]clsc_connect: (0x8605510) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_oracl-experts.info2_crs))

Investigation:

I looked into the files under /var/tmp/.oracle in node1 - Looks everything fine but in the 2nd node , the CRS process was not able to write

Solution:

After long research, I found out that there is a major issue with the /var/tmp

[root@oracl-experts.info2 app]# chmod 1777 /var/tmp [root@oracl-experts.info2 app]# ls -ld /var/tmp drwxrwxrwt 3 root root 4096 Oct 26 17:33 /var/tmp

total 8

drwxrwxrwt 3 root root 4096 Oct 26 17:33 ..
srwxrwxrwx 1 oracle oinstall 0 Oct 26 17:52 soracl-experts.info2DBG_CSSD
srwxrwxrwx 1 oracle oinstall 0 Oct 26 17:52 sOracle_CSS_LclLstnr_crs_2
srwxrwxrwx 1 oracle oinstall 0 Oct 26 17:52 sOCSSD_LL_oracl-experts.info2_crs
srwxrwxrwx 1 root root 0 Oct 26 17:52 sprocr_local_conn_0_PROC
srwxrwxrwx 1 oracle oinstall 0 Oct 26 17:52 soracl-experts.info2DBG_EVMD
srwxrwxrwx 1 root root 0 Oct 26 17:52 soracl-experts.info2DBG_CRSD
srwxrwxrwx 1 oracle oinstall 0 Oct 26 17:52 sSYSTEM.evm.acceptor.auth
srwxrwxrwx 1 oracle oinstall 0 Oct 26 17:52 sCoracl-experts.info2_crs_evm
srwxrwxrwx 1 oracle oinstall 0 Oct 26 17:52 sAoracl-experts.info2_crs_evm
srwxrwxrwx 1 root root 0 Oct 26 17:52 sora_crsqs
srwxrwxrwx 1 root root 0 Oct 26 17:52 sCRSD_UI_SOCKET
drwxrwxrwt 2 root root 4096 Oct 26 17:52 .

[root@oracl-experts.info2 app]# ./root.sh
WARNING: directory '/data01/crs/oracle/product/10.2.0' is not owned by root
WARNING: directory '/data01/crs/oracle/product' is not owned by root
WARNING: directory '/data01/crs/oracle' is not owned by root
WARNING: directory '/data01/crs' is not owned by root
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory

Setting up NS directories

Oracle Cluster Registry configuration upgraded successfully

WARNING: directory '/data01/crs/oracle/product/10.2.0' is not owned by root

WARNING: directory '/data01/crs/oracle/product' is not owned by root

WARNING: directory '/data01/crs/oracle' is not owned by root

WARNING: directory '/data01/crs' is not owned by root

clscfg: EXISTING configuration version 3 detected.

clscfg: version 3 is 10G Release 2.

assigning default hostname oracl-experts.info1 for node 1.

assigning default hostname oracl-experts.info2 for node 2.

Successfully accumulated necessary OCR keys.

Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.

node <nodenumber>: <nodename> <private interconnect name> <hostname>

node 1: oracl-experts.info1 oracl-experts.info1-priv oracl-experts.info1

node 2: oracl-experts.info2 oracl-experts.info2-priv oracl-experts.info2

clscfg: Arguments check out successfully.

NO KEYS WERE WRITTEN. Supply -force parameter to override.

-force is destructive and will destroy any previous cluster

configuration.

Oracle Cluster Registry for cluster has already been initialized

Startup will be queued to init within 90 seconds.

Adding daemons to inittab

Expecting the CRS daemons to be up within 600 seconds.

CSS is active on these nodes.

oracl-experts.info1

oracl-experts.info2

CSS is active on all nodes.

Waiting for the Oracle CRSD and EVMD to start

Oracle CRS stack installed and running under init(1M)

Running vipca(silent) for configuring nodeapps

The given interface(s), "eth0" is not public. Public interfaces should be used t o configure virtual IPs.

Cluster Split based on the service name

Cluster split - 2 nodes for DSS and 1 Node for OLTP

The requirement is to allocate 1 node exclusively to OLTP enviroment out of 3 node cluster.

The logic is based on the setting up the service_name

SQL> show parameter service_name

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
service_names string orclexperts.info
SQL>

SQL> alter system set service_names='OLTP_experts' sid='ORCLEXPERTS3';

System altered.

SQL>
[oracle@s262381ch3el101 ~]$ lsnrctl services | grep -i orclexperts
Service "ORCLEXPERTS.info" has 3 instance(s).
Instance "ORCLEXPERTS1", status READY, has 2 handler(s) for this service...
Instance "ORCLEXPERTS2", status READY, has 1 handler(s) for this service...
Instance "ORCLEXPERTS3", status READY, has 1 handler(s) for this service...
Service "ORCLEXPERTSXDB.info" has 3 instance(s).
Instance "ORCLEXPERTS1", status READY, has 1 handler(s) for this service...
Instance "ORCLEXPERTS2", status READY, has 1 handler(s) for this service...
Instance "ORCLEXPERTS3", status READY, has 1 handler(s) for this service...
Service "ORCLEXPERTS_XPT.info" has 3 instance(s).
Instance "ORCLEXPERTS1", status READY, has 2 handler(s) for this service...
Instance "ORCLEXPERTS2", status READY, has 1 handler(s) for this service...
Instance "ORCLEXPERTS3", status READY, has 1 handler(s) for this service...
Service "OLTP_ORCLEXPERTS.info" has 1 instance(s).
Instance "ORCLEXPERTS3", status READY, has 1 handler(s) for this service...

[oracle@experts1 admin]$ sqlplus system/xxxx@OLTP_ORCLEXPERTS

SQL*Plus: Release 10.2.0.2.0 - Production on Wed Feb 6 22:35:44 2008

Copyright (c) 1982, 2005, Oracle. All Rights Reserved.

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options

SQL>

SQL> show parameter service_name

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
service_names string OLTP_ORCLEXPERTS, ORCLEXPERTS.info

TNS:
OLTP_ORCLEXPERTS =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = orclexperts3-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = OLTP_ORCLEXPERTS.info)
(INSTANCE_NAME = ORCLEXPERTS3)
)
)

All OLTP connection strings changed to use the above string value.