Exadata Cloud at Customer: Considerations for Backup & Recovery

Working in a project where the customer migrates from an on-premises environment to Exadata Cloud at Customer provides insights about the decisions to take and what to consider when using the Exadata Cloud at Customer environment. In a series of blogs I want to talk about the different aspects when going to the Exadata Cloud at Customer (ExaCC).

Let’s start with the most important task a DBA is responsible for: Backup & Recovery.

REMARK: This blog is about DB-Backups only. Backing up other important data like e.g. your wallets with the TDE-keys is not covered in this blog.

On the ExaCC we basically have 3 possibilities to make backups (and restore them):

1. Using the Console to create and maintain automatic backups
2. Configuring and customizing Backups with bkup_api
3. Schedule your own rman backups

REMARK: In general it’s advised to use the automatic features Oracle provides on the ExaCC out of the box. A good description on how to set this up with bkup_api has been provided here.

The first 2 methods are well documented by Oracle, but they may not be flexible enough:

– with method 1 a restore is possible only on CDB-level, not on PDB-level
– there’s no possibility to call a script after the backup automatically
– you are limited to the backup destinations NFS, Zero Data Loss Recovery Appliance, local Exadata Storage, public Cloud Object storage
– there is only a limited number of retention period options for the different destinations

I.e. for some customers the automatic backups are not the first choice. Hence it may be necessary to schedule backups manually on the ExaCC. The question is: How to do this? There are some restrictions:

– it’s not allowed to create a crontab as user oracle on ExaCC
– scheduling backups with the dbms_scheduler job-type backup_script requires additional scripts for monitoring and logging purposes. See MOS Note An Example to Schedule RMAN Backup Using 12c DBMS_SCHEDULER BACKUP_SCRIPT (Doc ID 2102623.1) on how to schedule backups from the DB-scheduler.

A method used in a project was to start backup-scripts from a remote server through ssh. The idea was to get the least loaded primary node and ssh to that node and start the backup-job with nohub in the background and exit. The host which submits the backups could e.g. be the Oracle Management Server (OMS), i.e. the Enterprise Manager/Cloud Control server.

To ease the setup, we used the DMK (Database Management Kit) Backup scripts provided by dbi services. The following backups should be scheduled per database:

– weekly inc0 to NFS
– daily inc1 to NFS
– archivelog backup every hour to NFS
– delete everything obsolete on NFS every day
– daily backup of backupsets to the tape library (all backupsets on NFS which haven’t been backed up to tape yet)
– once a month a longterm backup to the tape library

REMARK: One reason to run on ExaCC is data sovereignty. Hence in such environments you usually do not backup to the object storage on the public cloud.

The question then was on how to connect to the DB-servers as ssh is open only for the fixed VM-IP-addresses, but not for e.g. the scan- or VIP-addresses. I.e. we have to be able to connect to the cluster even if a node is down. We resolved it by writing a simple procedure in the root container of each database which returns the node-name and ORACLE-SID. By connecting to the DB with sqlplus over a DB-service we were sure to connect to a running node. The simple code fragment, which returns the host-name and instance-name (ORACLE_SID) of the running RAC-instance was:

create or replace procedure get_host_and_SID (connected_host out varchar2, connected_SID out varchar2) as
begin
   select host_name, instance_name into connected_host, connected_SID from v$instance;
end;
/

To get the host-name and SID from the DB server the follwing Bash-script snippet can be used:

MYPARAMS=$(sqlplus -S ${DB_USER_GET_HOST_SID}/${PWD_GET_HOST_SID}@${DB_CONNECT_STRING} << EOF
set heading off lines 200 pages 0 feed off
var conn_host varchar2(64);
var conn_sid varchar2(16);
set serveroutput on
begin
   get_host_and_sid(:conn_host, :conn_sid);
   dbms_output.put_line(:conn_host||' '||:conn_sid);
end;
/
exit
EOF
)

RC=$?
if [ $RC -ne 0 ]
then
   echo "Getting params from DB failed."
   exit 1
fi

MYHOSTNAME=$(echo $MYPARAMS | cut -d " " -f1)
MYSID=$(echo $MYPARAMS | cut -d " " -f2)

And finally submit the backup-job:

# submitting job with nohup in the background:
ssh -t opc@$MYHOSTNAME "sudo su - oracle -c 'nohup ${LOCAL_DIR}/dmk_ha/bin/check_${LOWER_PRIM_OR_STANDBY}.ksh $MYSID ${LOCAL_DIR}/dmk_dbbackup/bin/dmk_rman.ksh -s $MYSID -t ${DMK_TEMPLATE} -u always >>/tmp/RMAN_BACKUP_${MYSID}.log 2>&1 &'" >> $LOGFILE

Summary: There are several methods on how to backup your DBs on the ExaCC-system. If the provided methods provided by Oracle are not flexible enough then you may write your own backup-scripts. Keep in mind that an ExaCC-system has some restrictions. You cannot easily schedule cron-jobs or ssh to a VIP.