I have been recently deploying dbvisit version 9.0.10 on Standard Edition SE2 18.7. Graceful switchover failed with error : ORA-00600: Interner Fehlercode, Argumente: [17090], [], [], [], [], [], [], [], [], [],[], [] (DBD ERROR: OCIStmtExecute)

Problem description

Running a graceful switchover will fail with ORA-00600 when converting the standby database if Unified Auditing is enabled. Output would be the following one :

oracle@ODA01:/u01/app/dbvisit/standby/ [TESTDB] ./dbvctl -d TESTDB -o switchover
=============================================================
Dbvisit Standby Database Technology (9.0.10_0_g064b53e) (pid 15927)
dbvctl started on ODA01-replica: Thu May 7 16:22:28 2020
=============================================================
 
>>> Starting Switchover between ODA01-replica and SEERP1SOP010-replica
 
Running pre-checks ... done
Pre processing ... done
Processing primary ... done
Processing standby ... done
Converting standby ... failed
Performing rollback ... done
 
>>> Database on server ODA01-replica is still a Primary Database
>>> Database on server SEERP1SOP010-replica is still a Standby Database
 
 
<<<>>>
PID:15927
TRACEFILE:15927_dbvctl_switchover_TESTDB_202005071622.trc
SERVER:ODA01-replica
ERROR_CODE:1
Remote execution error on SEERP1SOP010-replica.
 
==============Remote Output start: SEERP1SOP010-replica===============
Standby file
/u02/app/oracle/oradata/TESTDB_DC13/TESTDB_DC13/datafile/o1_mf_system_h5bcgo03_.dbf renamed to /u02/app/oracle/oradata/TESTDB_DC41/TESTDB_DC41/datafile/o1_mf_system_hbzpn4l6_.dbf in
database TESTDB.
Standby file
...
...
...
/u02/app/oracle/oradata/TESTDB_DC41/TESTDB_DC41/datafile/o1_mf_ts_edc_d_hbzpp9yw_.dbf in database TESTDB.
Standby file /u04/app/oracle/redo/TESTDB/TESTDB_DC13/onlinelog/o1_mf_2_h5bfq0xq_.log
renamed to /u01/app/dbvisit/standby/gs/TESTDB/X.DBVISIT.REDO_2.LOG in database TESTDB.
<<<>>>
PID:36409
TRACEFILE:36409_dbvctl_f_gs_convert_standby_TESTDB_202005071626.trc
SERVER:SEERP1SOP010-replica
ERROR_CODE:600
ORA-00600: Interner Fehlercode, Argumente: [17090], [], [], [], [], [], [], [], [], [],[], [] (DBD ERROR: OCIStmtExecute)
>>>> Dbvisit Standby terminated <<<>>> Dbvisit Standby terminated <<<<

Troubleshooting

The problem is known by Dbvisit. Their engineering team is already working on a fix that is planned to be released in version 9.1.XX of Dbvisit.

Workaround would be to disable Unified Auditing.

Disable Unified Auditing

In this part I will describe how to disable Unified Auditing. It is an ODA so my environment is using Oracle Restart.

1- Shutdown the database

Database shutdown is executed with Oracle user using srvctl :
oracle@ODA01:/u01/app/dbvisit/standby/ [rdbms18000_1] srvctl stop database -d TESTDB_DC13

2- Shutdown listener

Listener is own by grid user and stoping the listener will be executed using srvctl :
grid@ODA01:/home/grid/ [+ASM1] srvctl stop listener -listener LISTENER

3- Relink oracle executable

For the current database home, the oracle executable needs to be relinked :
oracle@ODA01:/u01/app/dbvisit/standby/ [rdbms18000_1] cd $ORACLE_HOME/rdbms/lib
 
oracle@ODA01:/u01/app/oracle/product/18.0.0.0/dbhome_1/rdbms/lib/ [rdbms18000_1] make -f ins_rdbms.mk uniaud_off ioracle ORACLE_HOME=$ORACLE_HOME
/usr/bin/ar d /u01/app/oracle/product/18.0.0.0/dbhome_1/rdbms/lib/libknlopt.a kzaiang.o
/usr/bin/ar cr /u01/app/oracle/product/18.0.0.0/dbhome_1/rdbms/lib/libknlopt.a /u01/app/oracle/product/18.0.0.0/dbhome_1/rdbms/lib/kzanang.o
chmod 755 /u01/app/oracle/product/18.0.0.0/dbhome_1/bin
- Linking Oracle
rm -f /u01/app/oracle/product/18.0.0.0/dbhome_1/rdbms/lib/oracle
/u01/app/oracle/product/18.0.0.0/dbhome_1/bin/orald -o /u01/app/oracle/product/18.0.0.0/dbhome_1/rdbms/lib/oracle -m64 -z noexecstack -Wl,--disable-new-dtags -L/u01/app/oracle/product/18.0.0.0/dbhome_1/rdbms/lib/ -L/u01/app/oracle/product/18.0.0.0/dbhome_1/lib/ -L/u01/app/oracle/product/18.0.0.0/dbhome_1/lib/stubs/ -Wl,-E /u01/app/oracle/product/18.0.0.0/dbhome_1/rdbms/lib/opimai.o /u01/app/oracle/product/18.0.0.0/dbhome_1/rdbms/lib/ssoraed.o /u01/app/oracle/product/18.0.0.0/dbhome_1/rdbms/lib/ttcsoi.o -Wl,--whole-archive -lperfsrv18 -Wl,--no-whole-archive /u01/app/oracle/product/18.0.0.0/dbhome_1/lib/nautab.o /u01/app/oracle/product/18.0.0.0/dbhome_1/lib/naeet.o /u01/app/oracle/product/18.0.0.0/dbhome_1/lib/naect.o /u01/app/oracle/product/18.0.0.0/dbhome_1/lib/naedhs.o /u01/app/oracle/product/18.0.0.0/dbhome_1/rdbms/lib/config.o -ldmext -lserver18 -lodm18 -lofs -lcell18 -lnnet18 -lskgxp18 -lsnls18 -lnls18 -lcore18 -lsnls18 -lnls18 -lcore18 -lsnls18 -lnls18 -lxml18 -lcore18 -lunls18 -lsnls18 -lnls18 -lcore18 -lnls18 -lclient18 -lvsnst18 -lcommon18 -lgeneric18 -lknlopt -loraolap18 -lskjcx18 -lslax18 -lpls18 -lrt -lplp18 -ldmext -lserver18 -lclient18 -lvsnst18 -lcommon18 -lgeneric18 `if [ -f /u01/app/oracle/product/18.0.0.0/dbhome_1/lib/libavserver18.a ] ; then echo "-lavserver18" ; else echo "-lavstub18"; fi` `if [ -f /u01/app/oracle/product/18.0.0.0/dbhome_1/lib/libavclient18.a ] ; then echo "-lavclient18" ; fi` -lknlopt -lslax18 -lpls18 -lrt -lplp18 -ljavavm18 -lserver18 -lwwg `cat /u01/app/oracle/product/18.0.0.0/dbhome_1/lib/ldflags` -lncrypt18 -lnsgr18 -lnzjs18 -ln18 -lnl18 -lngsmshd18 -lnro18 `cat /u01/app/oracle/product/18.0.0.0/dbhome_1/lib/ldflags` -lncrypt18 -lnsgr18 -lnzjs18 -ln18 -lnl18 -lngsmshd18 -lnnzst18 -lzt18 -lztkg18 -lmm -lsnls18 -lnls18 -lcore18 -lsnls18 -lnls18 -lcore18 -lsnls18 -lnls18 -lxml18 -lcore18 -lunls18 -lsnls18 -lnls18 -lcore18 -lnls18 -lztkg18 `cat /u01/app/oracle/product/18.0.0.0/dbhome_1/lib/ldflags` -lncrypt18 -lnsgr18 -lnzjs18 -ln18 -lnl18 -lngsmshd18 -lnro18 `cat /u01/app/oracle/product/18.0.0.0/dbhome_1/lib/ldflags` -lncrypt18 -lnsgr18 -lnzjs18 -ln18 -lnl18 -lngsmshd18 -lnnzst18 -lzt18 -lztkg18 -lsnls18 -lnls18 -lcore18 -lsnls18 -lnls18 -lcore18 -lsnls18 -lnls18 -lxml18 -lcore18 -lunls18 -lsnls18 -lnls18 -lcore18 -lnls18 `if /usr/bin/ar tv /u01/app/oracle/product/18.0.0.0/dbhome_1/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo18 -lserver18"; fi` -L/u01/app/oracle/product/18.0.0.0/dbhome_1/ctx/lib/ -lctxc18 -lctx18 -lzx18 -lgx18 -lctx18 -lzx18 -lgx18 -lordimt -lclscest18 -loevm -lclsra18 -ldbcfg18 -lhasgen18 -lskgxn2 -lnnzst18 -lzt18 -lxml18 -lgeneric18 -locr18 -locrb18 -locrutl18 -lhasgen18 -lskgxn2 -lnnzst18 -lzt18 -lxml18 -lgeneric18 -lgeneric18 -lorazip -loraz -llzopro5 -lorabz2 -lipp_z -lipp_bz2 -lippdcemerged -lippsemerged -lippdcmerged -lippsmerged -lippcore -lippcpemerged -lippcpmerged -lsnls18 -lnls18 -lcore18 -lsnls18 -lnls18 -lcore18 -lsnls18 -lnls18 -lxml18 -lcore18 -lunls18 -lsnls18 -lnls18 -lcore18 -lnls18 -lsnls18 -lunls18 -lsnls18 -lnls18 -lcore18 -lsnls18 -lnls18 -lcore18 -lsnls18 -lnls18 -lxml18 -lcore18 -lunls18 -lsnls18 -lnls18 -lcore18 -lnls18 -lasmclnt18 -lcommon18 -lcore18 -ledtn18 -laio -lons -lfthread18 `cat /u01/app/oracle/product/18.0.0.0/dbhome_1/lib/sysliblist` -Wl,-rpath,/u01/app/oracle/product/18.0.0.0/dbhome_1/lib -lm `cat /u01/app/oracle/product/18.0.0.0/dbhome_1/lib/sysliblist` -ldl -lm -L/u01/app/oracle/product/18.0.0.0/dbhome_1/lib `test -x /usr/bin/hugeedit -a -r /usr/lib64/libhugetlbfs.so && test -r /u01/app/oracle/product/18.0.0.0/dbhome_1/rdbms/lib/shugetlbfs.o && echo -Wl,-zcommon-page-size=2097152 -Wl,-zmax-page-size=2097152 -lhugetlbfs`
rm -f /u01/app/oracle/product/18.0.0.0/dbhome_1/bin/oracle
mv /u01/app/oracle/product/18.0.0.0/dbhome_1/rdbms/lib/oracle /u01/app/oracle/product/18.0.0.0/dbhome_1/bin/oracle
chmod 6751 /u01/app/oracle/product/18.0.0.0/dbhome_1/bin/oracle
(if [ ! -f /u01/app/oracle/product/18.0.0.0/dbhome_1/bin/crsd.bin ]; then
getcrshome="/u01/app/oracle/product/18.0.0.0/dbhome_1/srvm/admin/getcrshome" ;
if [ -f "$getcrshome" ]; then
crshome="`$getcrshome`";
if [ -n "$crshome" ]; then
if [ $crshome != /u01/app/oracle/product/18.0.0.0/dbhome_1 ]; then
oracle="/u01/app/oracle/product/18.0.0.0/dbhome_1/bin/oracle";
$crshome/bin/setasmgidwrap oracle_binary_path=$oracle;
fi
fi
fi
fi
);

4- Start listener

With grid user, execute :
grid@ODA01:/home/grid/ [+ASM1] srvctl start listener -listener LISTENER

5- Start database

With oracle user, execute :
oracle@ODA01:/home/oracle/ [rdbms18000_2] srvctl start database -d TESTDB_DC13

6- Deactivate any existing Unified Auditing policies

oracle@ODA01:/home/oracle/ [SALESPRD] sqh
 
SQL> set line 300
SQL> column user_name format a20
SQL> column policy_name format a50
SQL> column entity_name format a50
SQL> select * from audit_unified_enabled_policies;
 
USER_NAME POLICY_NAME ENABLED ENABLED_OPTION ENTITY_NAME ENTITY_ SUC FAI
-------------------- -------------------------------------------------- ------- --------------- -------------------------------------------------- ------- --- ---
ALL USERS ORA_SECURECONFIG BY BY USER ALL USERS USER YES YES
ALL USERS ORA_LOGON_FAILURES BY BY USER ALL USERS USER NO YES
 
SQL> noaudit policy ORA_SECURECONFIG;
 
NOAUDIT wurde erfolgreich ausgefuhrt.
 
SQL> noaudit policy ORA_LOGON_FAILURES;
 
NOAUDIT wurde erfolgreich ausgefuhrt.
 
SQL> select * from audit_unified_enabled_policies;
 
Es wurden keine Zeilen ausgewahlt