Infrastructure at your Service

Here is how to apply the latest patch for your ODA. As usual the Oracle documentation is not 100% accurate. I applied this patch on 2 ODAs X7-2M previously deployed in 12.2.1.2.0, no intermediate patch was needed.

1) Download the patch

The patch number is 28216794. This patch will update the following components:  dcs (odacli), operating system, bios/firmwares, ilom, GI, dbhomes and databases.
Download and copy the patch to a temporary folder on the server, for example /opt/patch. You’ll have to be root to apply the patch.

2) Check the actual versions and free space on disk

First check the current version:

odacli describe-component

System Version
---------------
12.2.1.2.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       12.2.1.2.0            up-to-date
GI                                        12.2.0.1.171017       up-to-date
DB {
[ OraDB12201_home1 ]                      12.2.0.1.171017       up-to-date
[ OraDB11204_home1 ]                      11.2.0.4.171017       up-to-date
}
DCSAGENT                                  12.2.1.2.0            up-to-date
ILOM                                      4.0.0.28.r121827      4.0.0.22.r120818
BIOS                                      41017600              41017100
OS                                        6.8                   up-to-date

The “available version” stands for “what’s available in the ODA repository?”. As this ODA has never been patched, it seems that some of the components are already in newer versions than those provided in the initial deployment package. Registering a new patch will refresh this repository.

Check that folders /, /u01 and /opt have enough free GB to process (>=10GB).

3) Prepare the patch files

Previous patch was slightly different from the others because Oracle simply forgot to double zip the file 2 and 3 of the patch. Now this patch is back to a more classic update: unzip and register the 3 unzipped files (they are zip files, too).

cd /opt/patch
unzip p28216794_122140_Linux-x86-64_1of3.zip
unzip p28216794_122140_Linux-x86-64_2of3.zip
unzip p28216794_122140_Linux-x86-64_3of3.zip

odacli update-repository -f /tmp/oda-sm-12.2.1.4.0-180708-server1of3.zip
odacli update-repository -f /tmp/oda-sm-12.2.1.4.0-180708-server2of3.zip
odacli update-repository -f /tmp/oda-sm-12.2.1.4.0-180708-server3of3.zip

Updating the repository, as other tasks through odacli, will generate a job-id you can look at to confirm that everything is running fine:

odacli describe-job -i "4087f0f4-2582-4621-b75c-59c9915a0cb5"

Job details
----------------------------------------------------------------
                     ID:  4087f0f4-2582-4621-b75c-59c9915a0cb5
            Description:  Repository Update
                 Status:  Success
                Created:  July 19, 2018 10:32:53 AM CEST
                Message:  oda-sm-12.2.1.4.0-180703-server1of3.zip

Note: you can update the repository with a single update-repository command including the 3 files. But it’s better to do the update separately in case of a corrupted file.

New feature with this patch, you can check if your system is ready for patching with this command:

odacli create-prepatchreport -s -v 12.2.1.4.0
odacli describe-prepatchreport -i f45f9750-ec9b-411f-ba53-43205cb17f87

4) Update the dcs-agent

First, you’ll have to update the dcs-agent:

/opt/oracle/dcs/bin/odacli update-dcsagent -v 12.2.1.4.0

odacli describe-job -i "5cc9174e-bd7a-435d-aaff-0113e9ab01bc"

Job details
----------------------------------------------------------------
                     ID:  5cc9174e-bd7a-435d-aaff-0113e9ab01bc
            Description:  DcsAgent patching
                 Status:  Success
                Created:  July 19, 2018 10:36:18 AM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation                July 19, 2018 10:36:18 AM CEST      July 19, 2018 10:36:19 AM CEST      Success
dcs-agent upgrade                        July 19, 2018 10:36:19 AM CEST      July 19, 2018 10:36:19 AM CEST      Success

This update takes only few seconds. Check again the version and you will see a new 18c component:

odacli describe-component

System Version
---------------
12.2.1.4.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       12.2.1.2.0            12.2.1.4.0
GI                                        12.2.0.1.171017       12.2.0.1.180417
DB {
[ OraDB12201_home1 ]                      12.2.0.1.171017       12.2.0.1.180417
[ OraDB11204_home1 ]                      11.2.0.4.171017       11.2.0.4.180417
}
DCSAGENT                                  18.2.1.0.0            up-to-date
ILOM                                      4.0.0.28.r121827      4.0.2.20.b.r123704
BIOS                                      41017600              41021300
OS                                        6.8                   6.9
FIRMWARECONTROLLER                        QDV1RE0F              qdv1re14

5) Update the server

Updating the server will update OS, ILOM, BIOS, firmwares and the GI. Update process will assume that no additionnal packages have been installed. If you installed additionnal packages, please remove them as they can prevent the patch to apply correctly.

This part of the update is the longest one (about 1 hour), and will end with a reboot of the server.

odacli update-server -v 12.2.1.4.0

odacli describe-job -i "a2e296c2-1b3e-4ed1-a5cc-0cb7c4d23120"

Job details
----------------------------------------------------------------
                     ID:  a2e296c2-1b3e-4ed1-a5cc-0cb7c4d23120
            Description:  Server Patching
                 Status:  Success
                Created:  July 19, 2018 12:12:53 PM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation                July 19, 2018 12:12:53 PM CEST      July 19, 2018 12:12:53 PM CEST      Success
dcs-controller upgrade                   July 19, 2018 12:12:54 PM CEST      July 19, 2018 12:12:54 PM CEST      Success
Patch location validation                July 19, 2018 12:12:54 PM CEST      July 19, 2018 12:12:54 PM CEST      Success
dcs-cli upgrade                          July 19, 2018 12:12:54 PM CEST      July 19, 2018 12:12:54 PM CEST      Success
Creating repositories using yum          July 19, 2018 12:12:54 PM CEST      July 19, 2018 12:12:55 PM CEST      Success
Applying HMP Patches                     July 19, 2018 12:12:55 PM CEST      July 19, 2018 12:14:31 PM CEST      Success
Patch location validation                July 19, 2018 12:14:31 PM CEST      July 19, 2018 12:14:31 PM CEST      Success
oda-hw-mgmt upgrade                      July 19, 2018 12:14:31 PM CEST      July 19, 2018 12:14:31 PM CEST      Success
Creating repositories using yum          July 19, 2018 12:14:31 PM CEST      July 19, 2018 12:14:35 PM CEST      Success
Applying OS Patches                      July 19, 2018 12:14:35 PM CEST      July 19, 2018 12:16:18 PM CEST      Success
OSS Patching                             July 19, 2018 12:16:18 PM CEST      July 19, 2018 12:16:18 PM CEST      Success
Applying Firmware Disk Patches           July 19, 2018 12:16:18 PM CEST      July 19, 2018 12:16:23 PM CEST      Success
Applying Firmware Expander Patches       July 19, 2018 12:16:23 PM CEST      July 19, 2018 12:16:26 PM CEST      Success
Applying Firmware Controller Patches     July 19, 2018 12:16:26 PM CEST      July 19, 2018 12:16:30 PM CEST      Success
Checking Ilom patch Version              July 19, 2018 12:16:31 PM CEST      July 19, 2018 12:16:33 PM CEST      Success
Patch location validation                July 19, 2018 12:16:33 PM CEST      July 19, 2018 12:16:34 PM CEST      Success
Apply Ilom patch                         July 19, 2018 12:16:35 PM CEST      July 19, 2018 12:16:36 PM CEST      Success
Copying Flash Bios to Temp location      July 19, 2018 12:16:37 PM CEST      July 19, 2018 12:16:37 PM CEST      Success
Starting the clusterware                 July 19, 2018 12:16:57 PM CEST      July 19, 2018 12:16:58 PM CEST      Success
clusterware patch verification           July 19, 2018 12:16:58 PM CEST      July 19, 2018 12:17:00 PM CEST      Success
Patch location validation                July 19, 2018 12:17:00 PM CEST      July 19, 2018 12:17:03 PM CEST      Success
Opatch updation                          July 19, 2018 12:17:33 PM CEST      July 19, 2018 12:17:35 PM CEST      Success
Patch conflict check                     July 19, 2018 12:17:35 PM CEST      July 19, 2018 12:18:16 PM CEST      Success
clusterware upgrade                      July 19, 2018 12:18:16 PM CEST      July 19, 2018 12:33:07 PM CEST      Success
Updating GiHome version                  July 19, 2018 12:33:07 PM CEST      July 19, 2018 12:33:11 PM CEST      Success
preRebootNode Actions                    July 19, 2018 12:33:30 PM CEST      July 19, 2018 12:34:11 PM CEST      Success
Reboot Ilom                              July 19, 2018 12:34:11 PM CEST      July 19, 2018 12:34:11 PM CEST      Success

If there is a problem during the patching process (you forgot to remove an additionnal rpm or one of the patch file is missing for example), you can relaunch the patching and it will skip the already patched components.

Once this part of the patch is successfuly applied, check again the components:

odacli describe-component

System Version
---------------
12.2.1.4.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       12.2.1.4.0            up-to-date
GI                                        12.2.0.1.180417       up-to-date
DB {
[ OraDB12201_home1 ]                      12.2.0.1.171017       12.2.0.1.180417
[ OraDB11204_home1 ]                      11.2.0.4.171017       11.2.0.4.180417
}
DCSAGENT                                  18.2.1.0.0            up-to-date
ILOM                                      4.0.2.20.b.r123704    up-to-date
BIOS                                      41017600              41021300
OS                                        6.9                   up-to-date
FIRMWARECONTROLLER                        QDV1RE14              up-to-date

Note: at this time, BIOS is not yet updated. Don’t know why but check this later and it will be OK.

6) Patch the dbhomes

You now need to patch the dbhomes separately. First of all, list them:

odacli list-dbhomes

ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
90556d26-6756-4fed-9546-d44d55b6fc04     OraDB11204_home1     11.2.0.4.171017 (26609929, 26392168)     /u01/app/oracle/product/11.2.0.4/dbhome_1     Configured
18a9c067-3629-409d-9bae-60d27516c914     OraDB12201_home1     12.2.0.1.171017 (27020386, 26710464)     /u01/app/oracle/product/12.2.0.1/dbhome_1     Configured

Patch the first one. Target version is actually version of the patch. Remember that updating the ODA through a patch will never upgrade your database to a newer release, your 11gR2 databases will stay in 11.2.0.4 and 12cR1 databases will stay in 12.1.0.2 (if you’re using one or both of these versions). Only the PSU number will change (the fifth number which is actually a date).

odacli update-dbhome -i 90556d26-6756-4fed-9546-d44d55b6fc04 -v 12.2.1.4.0

odacli describe-job -i "c1abf083-d597-4673-b07b-d7cb79ec434a"

Job details
----------------------------------------------------------------
                     ID:  c1abf083-d597-4673-b07b-d7cb79ec434a
            Description:  DB Home Patching: Home Id is 90556d26-6756-4fed-9546-d44d55b6fc04
                 Status:  Success
                Created:  July 19, 2018 12:59:11 PM CEST
                Message:  WARNING::Failed to run the datapatch as db DB03_S2 is not registered with clusterware##WARNING::Failed to run the datapatch as db DB04_S2 is not registered with clusterware##WARNING::Failed to run the datapatch as db DB05_S2 is not r

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
clusterware patch verification           July 19, 2018 12:59:12 PM CEST      July 19, 2018 12:59:15 PM CEST      Success
Patch location validation                July 19, 2018 12:59:15 PM CEST      July 19, 2018 12:59:19 PM CEST      Success
Opatch updation                          July 19, 2018 12:59:39 PM CEST      July 19, 2018 12:59:40 PM CEST      Success
Patch conflict check                     July 19, 2018 12:59:40 PM CEST      July 19, 2018 12:59:49 PM CEST      Success
db upgrade                               July 19, 2018 12:59:49 PM CEST      July 19, 2018 1:01:22 PM CEST       Success
SqlPatch upgrade                         July 19, 2018 1:01:22 PM CEST       July 19, 2018 1:01:22 PM CEST       Success
SqlPatch upgrade                         July 19, 2018 1:01:22 PM CEST       July 19, 2018 1:01:22 PM CEST       Success
SqlPatch upgrade                         July 19, 2018 1:01:22 PM CEST       July 19, 2018 1:01:23 PM CEST       Success
SqlPatch upgrade                         July 19, 2018 1:01:23 PM CEST       July 19, 2018 1:01:23 PM CEST       Success
SqlPatch upgrade                         July 19, 2018 1:01:23 PM CEST       July 19, 2018 1:01:23 PM CEST       Success
SqlPatch upgrade                         July 19, 2018 1:01:23 PM CEST       July 19, 2018 1:01:23 PM CEST       Success
SqlPatch upgrade                         July 19, 2018 1:01:23 PM CEST       July 19, 2018 1:01:23 PM CEST       Success
SqlPatch upgrade                         July 19, 2018 1:01:23 PM CEST       July 19, 2018 1:01:24 PM CEST       Success
SqlPatch upgrade                         July 19, 2018 1:01:24 PM CEST       July 19, 2018 1:01:24 PM CEST       Success
SqlPatch upgrade                         July 19, 2018 1:01:24 PM CEST       July 19, 2018 1:01:24 PM CEST       Success
SqlPatch upgrade                         July 19, 2018 1:01:24 PM CEST       July 19, 2018 1:01:24 PM CEST       Success
SqlPatch upgrade                         July 19, 2018 1:01:24 PM CEST       July 19, 2018 1:01:24 PM CEST       Success
SqlPatch upgrade                         July 19, 2018 1:01:24 PM CEST       July 19, 2018 1:01:25 PM CEST       Success

Despite the overall Success status, some database could stay in previous version for some reason:
– database is a StandBy database : no update is possible on the dictionary (dictionary will be updated when the primary server will be patched)
– database is not correctly registered in the Oracle Cluster repository (if you create the database with odacli in instance-only mode: create-database -io – Probably a bug)

After applying the patch, please check the version of your dictionary on each database with this statement :

select ACTION_TIME, ACTION, VERSION, ID, BUNDLE_SERIES, COMMENTS from  DBA_REGISTRY_HISTORY;
ACTION_TIME                    ACTION       VERSION                  ID BUNDLE_SERIES   COMMENTS
------------------------------ ------------ ---------------- ---------- --------------- ------------------------------
23/07/18 20:25:26,765974       APPLY        11.2.0.4             180417 PSU             PSU 11.2.0.4.180417

Then update the next dbhome:

odacli update-dbhome -i 18a9c067-3629-409d-9bae-60d27516c914 -v 12.2.1.4.0

odacli describe-job -i "ef53a8a6-bd74-40f6-a338-343489d41a1c"

Job details
----------------------------------------------------------------
                     ID:  ef53a8a6-bd74-40f6-a338-343489d41a1c
            Description:  DB Home Patching: Home Id is 18a9c067-3629-409d-9bae-60d27516c914
                 Status:  Success
                Created:  July 19, 2018 1:20:20 PM CEST
                Message:  WARNING::Failed to run datapatch on db DB12TST Failed to run Utlrp script##WARNING::Failed to run the datapatch as db DB12DEV: is not registered with clusterware

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
clusterware patch verification           July 19, 2018 1:20:41 PM CEST       July 19, 2018 1:20:42 PM CEST       Success
Patch location validation                July 19, 2018 1:20:42 PM CEST       July 19, 2018 1:20:42 PM CEST       Success
Opatch updation                          July 19, 2018 1:20:42 PM CEST       July 19, 2018 1:20:42 PM CEST       Success
Patch conflict check                     July 19, 2018 1:20:42 PM CEST       July 19, 2018 1:20:42 PM CEST       Success
db upgrade                               July 19, 2018 1:20:42 PM CEST       July 19, 2018 1:20:42 PM CEST       Success
SqlPatch upgrade                         July 19, 2018 1:20:42 PM CEST       July 19, 2018 1:20:47 PM CEST       Success
SqlPatch upgrade                         July 19, 2018 1:20:47 PM CEST       July 19, 2018 1:20:47 PM CEST       Success
SqlPatch upgrade                         July 19, 2018 1:20:47 PM CEST       July 19, 2018 1:21:03 PM CEST       Success

As previously patched dbhome, this one also has warnings. Please check if each database is OK.

Check the /opt/oracle/dcs/log/dcs-agent.log for extended warning messages if needed, and then check the DB Version after the update of all dbhomes:

odacli list-dbhomes

ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
90556d26-6756-4fed-9546-d44d55b6fc04     OraDB11204_home1     11.2.0.4.180417 (27441052, 27338049)     /u01/app/oracle/product/11.2.0.4/dbhome_1     Configured
18a9c067-3629-409d-9bae-60d27516c914     OraDB12201_home1     12.2.0.1.180417 (27464465, 27674384)     /u01/app/oracle/product/12.2.0.1/dbhome_1     Configured

Jump to step 8 if your databases are OK.

7) Update the databases where datapatch failed

If for some reason datapatch was not applied on some of your databases and it should have been, you’ll have to do it manually:

For 12c :

su – oracle
. oraenv <<< DB12TST
cd $ORACLE_HOME/OPatch
./datapatch -verbose

For 11gR2 :

su – oracle
. oraenv <<< DB11TST
sqlplus / as sysdba
@?/rdbms/admin/catbundle.sql psu apply

8) Patch the storage

No update-storage is needed for this patch on this ODA.

9) Optional: deploy the latest db clone files

If you’ll never deploy a new dbhome this step is not necessary. If you will, or simply if you don’t know if you’ll have to do that later, download and register the new db clone files in the repository to be able to create a new dbhome at the same patch level than the one already deployed, for example:

cd /opt/patch

unzip p27119402_122140_Linux-x86-64.zip
Archive:  p27119402_122140_Linux-x86-64.zip
 extracting: odacli-dcs-12.2.1.4.0-180617-DB-12.2.0.1.zip
  inflating: README.txt

update-image --image-files odacli-dcs-12.2.1.4.0-180617-DB-12.2.0.1.zip
Unpacking will take some time,  Please wait...
Unzipping odacli-dcs-12.2.1.4.0-180617-DB-12.2.0.1.zip

10) Control the final version of the components

Now the patching is done. ±2 hours were needed, if everything is OK.

oodacli describe-component

System Version
---------------
12.2.1.4.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       12.2.1.4.0            up-to-date
GI                                        12.2.0.1.180417       up-to-date
DB {
[ OraDB12201_home1 ]                      12.2.0.1.180417       up-to-date
[ OraDB11204_home1 ]                      11.2.0.4.180417       up-to-date
}
DCSAGENT                                  18.2.1.0.0            up-to-date
ILOM                                      4.0.2.20.b.r123704    up-to-date
BIOS                                      41021300              up-to-date
OS                                        6.9                   up-to-date
FIRMWARECONTROLLER                        QDV1RE14              up-to-date

11) Optional: patch the SFP firmware

If you’re using SFP network interfaces on your ODA X7 (fiber connected network), you may encounter network troubles (lost of pings, failover not working correctly, and so on). SFP are not part of the ODA patching process, but there is a firmware update available for those kind of SFP on Oracle X7 server familly (including ODA). Please review note 2373070.1 for more information :

Using the onboard SFP28 ports on an ODA X7-2 server node (Doc ID 2373070.1)

 

Leave a Reply

Jérôme Dubar
Jérôme Dubar