One of the key features of the ODA (Oracle Database Appliance) is the ability to patch the entire stack every three months, the goal being to keep all the components up-to-date. Most of the customers won’t patch so often, but one patch a year is quite a good average. But when comes the time for patching, comes the time for anxiety for the DBA. And it’s totally justified.
Why ODA patching can eventually be a nightmare?
First of all, patching all the products is a complex operation. ODA is not a real appliance: it’s classic hardware composed of parts from various vendors, and with nearly standard software, including Linux, Grid Infrastructure, ASM, ACFS and database engines. And all these products need to be patched together. If you were allowed to patch the ODA components separately, it could last quite a long time. Yes, Oracle provides a single patch for the ODA, but it’s just a bundle of dozen of patches. It’s easier to apply, all together the patches are certified, but it’s still a complex operation to bring all the modules to the target patch level. This is why you can encounter multiple problems. For example if you installed your own RPMs onto the system (unable to update the OS), if you lack some free space (unable to complete the patching), if your databases have specific configuration, or eventually if you discover that there is a bug in the patch related to the version you come from and linked to your ODA model.
Also, some of the patches are not cumulative, meaning that you cannot directly upgrade to the latest version. You sometimes need to apply 4 or 5 patches to upgrade, making the patching even more uncertain.
Starting from these facts, you may think about reimaging, and you’re probably right.
What are the advantages and drawbacks of reimaging?
For sure, reimaging has a lot of advantages:
- Guarantee of success (you start from scratch)
- Cleaner ODA (no question about that)
- Make sure you are able to do the reimage (in case of you really need it)
- Make sure your documentation is good (and this is the only way to validate it!)
- Avoid later problems if patching not worked correctly
These are the drawbacks:
- Longer than a single patch succesfully applied on the first try (who knows)
- Need to erase everything and restart as if it were a new ODA
- You need to know how your ODA was installed and configured (not so simple if someone did the job for you)
- You probably need another ODA with Data Guard or DBVisit to limit the downtime
Can reimaging be quicker than patching?
Patching last about 3 hours if everything is OK. But it’s only for one patch and only if everything is OK. With my patching experience, you probably need to plan 1 day for the first ODA you will patch.
Reimaging also last about 3 hours (more or less depending on your ODA version). But it’s only for reinstalling the software without any database. You will need to restore all your databases, and do all the things you’ve done at the first deployment: copy your scripts, setup your additional software, restore your crontabs, your specific configuration, put back monitoring, and so on.
So, reimaging is probably longer, but you are quite sure to redeploy your ODA in a known time. This is a strong argument. “It will take 8 hours” is always better than “it would take between 3 and 8 hours. Or maybe more. If I succeed”.
How to proceed with patches?
If you need to patch regularly, try to apply the patch on a ODA you can live without. If something goes wrong, you can decide to reimage very quickly instead of opening a SR on MOS. Please don’t get stuck because a patch is not applying correctly, it’s a waste of time.
If you patch every year, consider redeploying instead of patching. It’s probably more work but it will take the same amount of time, with success guarantee (you will love that point). Also, you will ensure that you are able to reimage completely. Reimaging is sometimes also needed if you move your ODA to another datacenter with a network change, so you could have to reimage even for other reasons than patching.
How to make sure that you are able to reimage?
This is the key: be able to reimage
Rules to follow:
- restrict the access on your ODA to only people concerned about the appliance
- document every change you make on the server, even a simple chmod
- never use the GUI to deploy the appliance: deploy your ODA using odacli and save the deployment json file outside of the ODA
- never use the GUI to create the databases: create the database with odacli and backup the used parameters in the documentation
- use scripts to configure your databases (avoid one-shot changes)
- install other products only if necessary: do you really need a backup tool on ODA? NFS backups are great and easy to configure without installing anything
- install only RPMs manually from Oracle ISOs and only if needed
- do everything from the command line and avoid using vi. Text editors prevent you from being able to repeat the exact same operation. For example, replace
echo "srv-nfs:/orabackups /backup nfs rw,bg,hard,nolock,nointr" >> /etc/fstab
- always consider your ODA not so critical by having the possibility to restore your database elsewhere (understand on another ODA), or adopt Data Guard or DBVisit for all your databases that cannot support to be down for hours (even development databases are production for developpers!)
- keep the install zipfiles corresponding to your version somewhere secured to avoid searching for them on MOS the day you need to reimage
Regarding the scripts, I always create a scripts folder in /home/oracle on ODA, and each database has 3 dedicated scripts to speed up the database recreation if needed: create_SID.sh, configure_SID.sql and tbs_SID.sql. First script is for odacli database creation, first SQL script if for specific configuration (controlfile multiplexing for example, disabling the recycle bin or enabling the archive_lag_target, etc). Second SQL script is for tablespace creation. Target is to be able to recreate the database even for datapump-based restore. Make sure to backup these scripts somewhere else.
Few words about RPMs : for me the best way to install additional RPMs on ODA is to download the Oracle Linux ISO corresponding to the version on your ODA (the ISO you would use if you need to deploy a normal server), mount the ISO on your ODA and pickup only the RPMs you need from it (you can also put these few RPMs on /home/oracle/RPMs).
Reimaging should always be considered as an alternative way of patching. Or the best way. Companies already having integrated this are happier with their ODAs. And are taking the best from these appliances.