adrci failed with : Could not load program adrci:

I normally use adrci tool to find any ORA- Error in Alert log and report to DBA's Team if any Error message found in last n hours.

Script fails with following error on AIX environment whereas works fine in HP-UX. 

Error :

Could not load program adrci:
rtld: 0712-001 Symbol kgeres_where was referenced
      from module adrci(), but a runtime definition
     of the symbol was not found.
rtld: 0712-001 Symbol skge_sign_fr was referenced
      from module adrci(), but a runtime definition
     of the symbol was not found.
rtld: 0712-002 fatal error: exiting.

Problem : 

As indicated in Error message its not able load one of required library even though LD_LIBRARY_PATH and ORALE_HOME is set up correctly.

Solution:

You would need setting additional parameter LIBPATH for this tool to work.

Example : 

export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1

export LIBPATH=$ORACLE_HOME/lib

runcluvfy.sh hangs @user file creation mask or after "Checking to make sure user "oracle" is not in "root" group"

/runcluvfy.sh stage -pre crsinst -upgrade -n dbp2,dbp4 -rolling -src_crshome $ORACLE_HOME -dest_crshome /u01/app/product/11.2.0.3 -dest_version 11.2.0.3.0  -fixup -fixupdir /tmp -verbose


Performing pre-checks for cluster services setup

Checking node reachability...


...........................
.........................


Checking to make sure user "oracle" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  dbp4    passed                    does not exist
  dbp2    passed                    does not exist
Result: User "oracle" is not part of "root" group. Check passed

<<<<<--------- Hangs Here >>>>>>

Check default user file creation mask
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  dbp4    077                       0022                      failed
  dbp2    022                       0022                      passed

===========================

This issue happened many times and mostly caused by .profile and same was in my case. even though there are not many parameter/variable used in .profile.

Solution:

temporarily moving the $HOME/.profile file to temp name while running the test resolved the issue.  I will comment individual parameter sometime later to identify which particular one causing issue.

--------
Note:
Reviewing the CV LOG identified the problem. .profile was calling a interactive script which wait for user input.

Any issue we should first review the log under $ORA_CRS_HOME/cv/log (10g) / $GI_HOME/cv/log (11g onwards), they are very detailed and handy.

Thanks

krishan

ORA-15041 : diskgroup space exhausted

ORA-15041: diskgroup "DATA01" space exhausted: appeared on the database

ASMCMD-08016: copy source->'+DATA01/db/datafile/ts_db_summary_index005.dbf' and target->'+DATA01/db/datafile/ts_db_summary_index005.dbf' failed
ORA-19505: failed to identify file "+DATA01/db/datafile/ts_db_summary_index005.dbf"
ORA-17502: ksfdcre:4 Failed to create file +DATA01/db/datafile/ts_db_summary_index005.dbf
ORA-15041: diskgroup "DATA01" space exhausted
ORA-06512: at "SYS.X$DBMS_DISKGROUP", line 410
ORA-06512: at line 3 (DBD ERROR: OCIStmtExecute)
copying knudwdbp1:+DATA01/db/datafile/ts_db_summary_index006.dbf -> +DATA01/db/datafile/ts_db_summary_index006.dbf
Argument "normal" isn't numeric in numeric ge (>=) at /u01/app/product/grid/11.2.0/lib/asmcmdshare.pm line 1676.
errors

----------------------------------

Initial investigation indicated there is around 650GB free space in disk group but still failed to create 1GB file.

Detailed analysis of space usage revealed there wer 29 disks  in the diskgroup and 2 of disk were 100%. All disk are of same size but two still managed to filled 100%.

So even 27 disk still got 50% free space Oracle still error out ORA-15041.  Running the re balance operation managed to fix the issue.

I know we normally check freespace in diskgroup but sometime its worth checking free space in individual disk in diskgroup to ensure disk are full balanced in term of space utilization. 

Note: Oracle/ASM doesn't store data unevenly its must be manual error , when somebody stopped to Re Balance operation due to whatever reason and never run again.

Thanks

krishan

ORACLE RAC FAQ's

Q:- How to change Interconnect Protocol : 

1) Login as Oracle Grid Infrastructure owner
2) Setup oracle Grid HOME
3) Remake ioracle  library with new protocol

To change from default UDP to IPC , issue the following


Prompt> make -f  $ORACLE_HOME/rdbms/lib/ins_rdbms.mk ipc_rds ioracle

Q:-