Infrastructure at your Service

Introduction

Patching an Oracle Database Appliance is sometimes quite tough and you often struggle to solve failures at different steps. Regarding ILOM and BIOS patching errors, a simple workaround is possible.

Errors when patching ILOM and/or BIOS

This is something I recently had on an ODA X7-2S when trying to apply patch 19.11. Patching of the test server was OK, but patching the production server on a week-end was not OK. I needed to find a solution quickly:

odacli update-server -v 19.11.0.0.0

Job details
----------------------------------------------------------------
                     ID:  fa8d3178-b12e-4bef-926e-437536e71582
            Description:  Server Patching
                 Status:  Failure
                Created:  August 7, 2021 2:07:18 PM CEST
                Message:  DCS-10001:Internal error encountered: Failed update Ilom using USB.

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Server patching                          August 7, 2021 2:07:32 PM CEST      August 7, 2021 2:11:35 PM CEST      Failure
Server patching                          August 7, 2021 2:07:33 PM CEST      August 7, 2021 2:11:35 PM CEST      Failure
Patch location validation                August 7, 2021 2:07:34 PM CEST      August 7, 2021 2:07:34 PM CEST      Success
dcs-controller upgrade                   August 7, 2021 2:07:35 PM CEST      August 7, 2021 2:07:35 PM CEST      Success
Creating repositories using yum          August 7, 2021 2:07:35 PM CEST      August 7, 2021 2:07:39 PM CEST      Success
Updating YumPluginVersionLock rpm        August 7, 2021 2:07:39 PM CEST      August 7, 2021 2:07:39 PM CEST      Success
Applying OS Patches                      August 7, 2021 2:07:39 PM CEST      August 7, 2021 2:07:40 PM CEST      Success
Creating repositories using yum          August 7, 2021 2:07:40 PM CEST      August 7, 2021 2:07:40 PM CEST      Success
Applying HMP Patches                     August 7, 2021 2:07:40 PM CEST      August 7, 2021 2:07:41 PM CEST      Success
Patch location validation                August 7, 2021 2:07:41 PM CEST      August 7, 2021 2:07:41 PM CEST      Success
oda-hw-mgmt upgrade                      August 7, 2021 2:07:42 PM CEST      August 7, 2021 2:07:42 PM CEST      Success
OSS Patching                             August 7, 2021 2:07:42 PM CEST      August 7, 2021 2:07:42 PM CEST      Success
Applying Firmware Disk Patches           August 7, 2021 2:07:42 PM CEST      August 7, 2021 2:07:45 PM CEST      Success
Applying Firmware Controller Patches     August 7, 2021 2:07:46 PM CEST      August 7, 2021 2:07:48 PM CEST      Success
Server patching                          August 7, 2021 2:07:48 PM CEST      August 7, 2021 2:11:35 PM CEST      Failure
Checking Ilom patch Version              August 7, 2021 2:07:49 PM CEST      August 7, 2021 2:07:49 PM CEST      Success
Patch location validation                August 7, 2021 2:07:49 PM CEST      August 7, 2021 2:07:49 PM CEST      Success
Save password in Wallet                  August 7, 2021 2:07:50 PM CEST      August 7, 2021 2:07:50 PM CEST      Success
Server patching                          August 7, 2021 2:07:50 PM CEST      August 7, 2021 2:11:35 PM CEST      Failure
Apply Ilom patch                         August 7, 2021 2:07:50 PM CEST      August 7, 2021 2:11:34 PM CEST      Failure

Actually, patching the ILOM is asking for port 623 to be opened between the node and the ILOM, but you probably won’t open it because it’s not used during normal operation. This was the cause of my problem. On my test server, ILOM and node were on the same network, they were on different networks on production. So how could you do to solve this problem?

Workaround

As you may know, ODA is not more than a classic server from Oracle’s x86 lineup. This means that if you would like to install Windows Server on your ODA, it would be possible without any trick. It also means that manual patching process for microcodes exists for this kind of server. Do the ILOM and BIOS need to be patched with odacli? Yes, but you can also apply the patch separately, odacli has no specific procedure for patching these components. They are not related to GI or DB, so patching manually should be OK.

You will find the procedure for patching ILOM and BIOS in the documentation of these servers’ lineup. Actually, this is a very simple operation, because you just need to upload the ILOM and BIOS package to the ILOM web interface, and patch from there. The package will be easy to find because as you planned to patch your ODA, the patch is already available in the repository. This example comes from a X8-2M ODA:

ls -lrt /opt/oracle/oak/pkgrepos/ilom/x8-2l/5.0.1.21.a.r138015/
total 38092
-rwxrwxr-x 1 root root      347 May 31 03:50 componentmetadata.xml
-rwxrwxr-x 1 root root 38999414 May 31 03:50 ILOM-5_0_1_21_a_r138015-ORACLE_SERVER_X8-2L-rom.pkg

The .pkg file is a package for BIOS and ILOM.

My X8-2M ODA is up to date for most components but not for ILOM:

[[email protected] dbi]# odacli describe-component
System Version
---------------
19.11.0.0.0

System node Name
---------------
dbi-oda-x8

Local System Version
---------------
19.11.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.11.0.0.0           up-to-date
GI                                        19.11.0.0.210420      up-to-date
DB {
[ OraDB19000_home1 ]                      19.11.0.0.210420      up-to-date
[ OraDB18000_home1 ]                      18.14.0.0.210420      up-to-date
}
DCSAGENT                                  19.11.0.0.0           up-to-date
OS                                        7.9                   up-to-date
ILOM                                      5.0.1.21.r136383      5.0.1.21.a.r138015
BIOS                                      52030400              up-to-date
FIRMWARECONTROLLER                        VDV1RL04              up-to-date
FIRMWAREDISK                              1132                  up-to-date
HMP                                       2.4.8.0.600           up-to-date

Let’s apply the ILOM patch from the ILOM itself.

In the ILOM Administration menu, click on Maintenance. Then Enter Update Mode (don’t be afraid of the warning):


Upload the .pkg file here:



Upload is in progress:

This compares what’s on your server and what’s in the package. For sure, you want to preserve ILOM and/or BIOS settings. Then click on Start Update:

This is obvious, but don’t unplug your ODA during this patching!

Patch is applying:



This is now finished, it only took a few minutes.

After this update, ILOM will restart.


Doing another describe-component now shows that ILOM has been updated:

odacli describe-component
System Version
---------------
19.11.0.0.0

System node Name
---------------
dbi-oda-x8

Local System Version
---------------
19.11.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.11.0.0.0           up-to-date
GI                                        19.11.0.0.210420      up-to-date
DB {
[ OraDB19000_home1 ]                      19.11.0.0.210420      up-to-date
[ OraDB18000_home1 ]                      18.14.0.0.210420      up-to-date
}
DCSAGENT                                  19.11.0.0.0           up-to-date
OS                                        7.9                   up-to-date
ILOM                                      5.0.1.21.a.r138015    up-to-date
BIOS                                      52030400              up-to-date
FIRMWARECONTROLLER                        VDV1RL04              up-to-date
FIRMWAREDISK                              1132                  up-to-date
HMP                                       2.4.8.0.600           up-to-date

If needed, BIOS will be updated too as it’s part of the package, but it will trigger a node reboot, so be careful. You can postpone the BIOS update after your next reboot of the node if you prefer.

If your ODA is an HA model, you’ll need to apply this package on both nodes.

Conclusion

Don’t waste your time on this kind of problem. ILOM and BIOS patching are basic operations you can manually do when odacli doesn’t succeed. But patch only from your existing ODA repository, packages coming from other sources could be hazardous.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Jérôme Dubar
Jérôme Dubar

Senior Consultant