Infrastructure at your Service

Introduction

One of the advantage of Oracle Database Appliance is its fast deployment. Most often, the initial setup of a lite ODA (with a reimaging) only takes half a day, from unboxing until a first test database is available. Trust me, it’s hard to do better. For some reason, for example if you need to patch to the latest release and you have some intermediate patch to apply before, or if you plan to change your network settings, or if your ODA does not work correctly anymore, you can imagine redeploying your ODA from scratch as it will not take days. But as you may know, redeployment means that everything will be erased, and you will have to rebuild all your databases, apply your various settings again, and so on. As a consequence, redeploying an ODA is only something you will consider if you use Data Guard or Dbvisit Standby in order to switch your databases to another ODA before redeployment. A standalone ODA is never a comfortable solution, and you’ll probably never be able to redeploy it.

Appart from your DBA job, redeploying an ODA is split in 2 tasks: reimaging the system, and creating the appliance. Reimaging is basically an OS installation that will erase the system disks, creating the appliance is installing all the Oracle stack, from Linux user creation until a first database creation.

But how much time do you need to redeploy an ODA? Let’s find out some clues based on a redeployment I did a few days ago.

Preparing redeployment

A good preparation is mandatory in order to minimize the downtime due to redeployment. Even if you use Data Guard or Dbvisit Standby, you’d better do the redeployment in few hours instead of few days because you will not be able to failover in case of a problem during this operation (and if you’re very unlucky).

What you need to prepare before starting:

  • all the files needed for redeployment according to your target version: ISO file, patch file, GI clone, RDBMS clones
  • unzip the ISO file for reimaging on your own computer (you don’t need to unzip the other files)
  • connect to ILOM interface to make sure that you still have the credentials and that remote console is working
  • backup each important file on an external volume in order to recover some parameters or files if needed

Important files for me are:

  • the result of odacli describe-system
  • history of root commands
  • history of oracle commands
  • crontab of oracle user
  • crontab of root user
  • content of fstab file
  • content of oratab file
  • list of all running databases ps -ef | grep pmon
  • list of all configured databases odacli list-databases
  • deployment file used for initial deployment of this ODA
  • or if deployment file is missing, information related to network configuration, disk sizing, redundancy, aso
  • tnsnames.ora, sqlnet.ora and listener.ora from the various homes

This list is absolutely not complete as it depends on your own environment and tools. Take enough time to be sure you miss nothing. And do this backup days before redeployment.

Cleanup data disks

Reimaging of an ODA will not care about existing headers on NVMe data disks. And will work as if the disks were empty. But odacli create-appliance cares and will prevent Grid Infrastructure to initialize the disks. And appliance creation will fail.

To avoid this behavior, the first step before starting the reimaging is to do a cleanup of the ODA. On an old ODA with spinning disks, it can take a while (many hours), but on a X8-2S or X8-2M it’s a matter of minutes. The longest operation being actually waiting for the node to reboot after cleanup.

My redeployment day started at 9.15AM. The target is to redeploy a X8-2M ODA, already running the latest 19.9 but with a new network configuration to apply. I did a backup of configuration files and already prepared newest json file for redeployment (I just had to change the network settings).

9.15AM => 9.25AM
Do the cleanup with /opt/oracle/oak/onecmd/cleanup.pl

Redeployment

First step of redeployment is reimaging the server with the ISO file corresponding to the target version.

9.25AM => 9.30AM
Start the remote console on ILOM, connect the ISO file in the storage menu, change boot options and restart the server from the CDROM

9.30AM => 10.15AM
Reimaging is automatic and quite long, after being sure the reimaging process is starting, it’s already time for a coffee

10.15AM => 10.20AM
Start configure-firstnet, the very basic initial network configuration (public IP, netmask and gateway). TFA restart is slowing down the operation.

10.20AM => 10.25AM
Copy the zipfiles from your computer to the server (I usually create a /opt/customer folder for these files): the GI clone and all the DB clones you will need

10.25AM => 10.30AM
Unzip and register in the ODA repository GI clone and first DB clone (I usually deploy the DB clone corresponding to the GI clone version)

10.30AM => 11.15AM
odacli create-appliance with your json file. Always use a json file for the create-appliance, GUI is slower and you could make mistakes during configuration.

11.15AM => 11.20AM
Configure the license with odacli update-cpucore -c x. Reimaging will enable again all the cores on your ODA, be careful about that. When using SE2, you do not need to decrease the number of cores, but I highly recommend you to do so.

11.20AM => 11.35AM
Unzip and install the other dbhomes (this is the time it took to register 3 more dbclones and create for each one a dbhome)

11.35AM => 11.50AM
Sanity checks: usually I do a describe-system and describe-components to see if everything is OK, I also check ASM disgroups available space and running instances (+ASM1, +APX1 and my DBTEST)

11.50AM => 12.10PM
It’s now time to configure all the system stuff: extend /opt and/or /u01 to your needs, declare nfs mountpoints (/etc/fstab), add networks with odacli create-network, set mtu on these networks if required, deploy the DBA tools and scripts, set the hugepages to a higher value (from 50% to 70% most often for me) and do a reboot of the server. Now I can enjoy lunch time and I’m quite confident for the next steps because everything looks fine on the pure ODA side.

It took me 3 hours to do the job, let’s say 4 hours if you’re not used to do that. It’s not so long compared to a single patch duration (about 3 hours).

Next steps

For sure, redeploying your ODA is just a (small) part of the job. You now have to recreate/restore your databases, or duplicate them again. And it takes time, especially if you have big databases and a limited number of cores. Never forget that SE2 is not able to parallelize operations like a restore, and EE is limited by the number of cores you enabled on your ODA (2 cores is one processor license).

The database tasks is most probably the biggest part: you may spend hours or days to rebuild your complete server. But with good procedures and good scripts, this task could finally be quite interesting.

If you experience problems during this part, you will learn how to solve them and improve your procedures for the other ODAs.

Advice

I strongly recommend to document everything you do on your ODAs, and script what you can script. And not just before a redeployment. Do that from the very moment you start working with ODA. You will never be affraid of a redeployment if you know exactly how your ODA was installed.

Conclusion

As I already concluded, it makes definitely sense to consider redeployment as an alternative way of patching, and much more. If you know how to quickly redeploy, it could be very helpful compared to days of complex troubleshooting for example. Keep in mind this possibility, even if you’d never consider this option on another platform.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Jérôme Dubar
Jérôme Dubar

Senior Consultant