Infrastructure at your Service

The ODA software 19.6 is released and people are starting to patch.
A direct patch to version 19.6 is possible from version 18.8.
Before patching your deployment to Oracle Database Appliance release 19.6, you must upgrade the operating system to Oracle Linux 7.
In this blog I am describing the steps I follow when patching an ODA from 18.8 to 19.6. I am using an ODA X7-2 (one node)
The first step is of course to download the required patch. You will need to copy following patch to your ODA.
Patch 31010832 : ORACLE DATABASE APPLIANCE 19.6.0.0.0 SERVER PATCH FOR ALL ODACLI/DCS STACK
Patch 30403662 : ORACLE DATABASE APPLIANCE RDBMS CLONE FOR ODACLI/DCS STACK

The first patch 31010832 will be used to upgrade the OS to Linux 7 and to patch you server to Appliance 19.6
The patch 30403662 will install the Oracle 19c clone database

The first patch contains 4 files

p31010832_196000_Linux-x86-64_1of4.zip
p31010832_196000_Linux-x86-64_2of4.zip
p31010832_196000_Linux-x86-64_3of4.zip
p31010832_196000_Linux-x86-64_4of4.zip

Just use the unzip command to decompress

[email protected]:/u01/oda_patch_mdi/19.6/ [ORCL] unzip p31010832_196000_Linux-x86-64_1of4.zip
[email protected]:/u01/oda_patch_mdi/19.6/ [ORCL] unzip p31010832_196000_Linux-x86-64_2of4.zip
[email protected]:/u01/oda_patch_mdi/19.6/ [ORCL] unzip p31010832_196000_Linux-x86-64_3of4.zip
[email protected]:/u01/oda_patch_mdi/19.6/ [ORCL] unzip p31010832_196000_Linux-x86-64_4of4.zip

Once done we have following files that we will use to update the repository

oda-sm-19.6.0.0.0-200420-server1of4.zip
oda-sm-19.6.0.0.0-200420-server2of4.zip
oda-sm-19.6.0.0.0-200420-server3of4.zip
oda-sm-19.6.0.0.0-200420-server4of4.zip

After you will have to list your scheduled jobs with the list-schedules option

[[email protected]]# odacli list-schedules

ID                                       Name                      Description                                        CronExpression                 Disabled
---------------------------------------- ------------------------- -------------------------------------------------- ------------------------------ --------
6d9cd445-8890-4bd6-a713-f6eb8fce35d0     metastore maintenance     internal metastore maintenance                     0 0 0 1/1 * ? *                true   
113ea801-636c-45d9-ad70-448054d825d5     AgentState metastore cleanup internal agentstateentry metastore maintenance     0 0 0 1/1 * ? *                false
e1780bf3-2467-4a89-9298-857fed7fa101     bom maintenance           bom reports generation                             0 0 1 ? * SUN *                false  
2f39ed40-6fc9-4547-b9f1-11889c5c7df9     Big File Upload Cleanup   clean up expired big file uploads.                 0 0 1 ? * SUN *                false  
a624a5c4-4801-444b-9b48-8be99f9f9e48     feature_tracking_job      Feature tracking job                               0 0 20 ? * WED *               false  

And disable all enabled jobs (column disabled to false). This done by using the command update-server with -d for disable and -i for the job_id

[[email protected]~]# odacli update-schedule -d -i  113ea801-636c-45d9-ad70-448054d825d5
Update job schedule success
[[email protected]~]# odacli  update-schedule -d -i  e1780bf3-2467-4a89-9298-857fed7fa101
Update job schedule success
[[email protected]~]# odacli  update-schedule -d -i  2f39ed40-6fc9-4547-b9f1-11889c5c7df9
Update job schedule success
[[email protected]~]# odacli  update-schedule -d -i  a624a5c4-4801-444b-9b48-8be99f9f9e48
Update job schedule success
[[email protected]~]#

When listing the jobs again, the column disabled must return true for all jobs

odabr is required. for the patching to 19.6. odabr is a tool to backup and recover an ODA. When running the precheck. the result will return failed if odabr is not installed. Please you can consult following document for downloading and using odabr : ODA (Oracle Database Appliance): ODABR a System Backup/Restore Utility (Doc ID 2466177.1)
By default odabr requires 190G free space in the LVM which can be not the case sometimes. In such case odabr should be run using some specific options. In my case I used the command below to take a backup of ODA.

[[email protected]~]# /opt/odabr/odabr backup -snap -osize 40 -rsize 20 -usize 80
INFO: 2020-05-29 09:31:15: Please check the logfile '/opt/odabr/out/log/odabr_81097.log' for more details


│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│
 odabr - ODA node Backup Restore - Version: 2.0.1-55
 Copyright Oracle, Inc. 2013, 2020
 --------------------------------------------------------
 Author: Ruggero Citton 
 RAC Pack, Cloud Innovation and Solution Engineering Team
│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│

INFO: 2020-05-29 09:31:15: Checking superuser
INFO: 2020-05-29 09:31:15: Checking Bare Metal
INFO: 2020-05-29 09:31:15: Removing existing LVM snapshots
WARNING: 2020-05-29 09:31:15: LVM snapshot for 'opt' does not exist
WARNING: 2020-05-29 09:31:15: LVM snapshot for 'u01' does not exist
WARNING: 2020-05-29 09:31:16: LVM snapshot for 'root' does not exist
INFO: 2020-05-29 09:31:16: Checking LVM size
INFO: 2020-05-29 09:31:16: Boot device backup
INFO: 2020-05-29 09:31:16: ...getting boot device
INFO: 2020-05-29 09:31:16: ...making boot device backup
INFO: 2020-05-29 09:31:20: ...boot device backup saved as '/opt/odabr/out/hbi/boot.img'
INFO: 2020-05-29 09:31:21: ...boot device backup check passed
INFO: 2020-05-29 09:31:21: Getting EFI device
INFO: 2020-05-29 09:31:21: ...making efi device backup
INFO: 2020-05-29 09:31:24: ...EFI device backup saved as '/opt/odabr/out/hbi/efi.img'
INFO: 2020-05-29 09:31:24: ...EFI device backup check passed
INFO: 2020-05-29 09:31:24: OCR backup
INFO: 2020-05-29 09:31:26: ...ocr backup saved as '/opt/odabr/out/hbi/ocrbackup_81097.bck'
INFO: 2020-05-29 09:31:26: Making LVM snapshot backup
SUCCESS: 2020-05-29 09:31:28: ...snapshot backup for 'opt' created successfully
SUCCESS: 2020-05-29 09:31:29: ...snapshot backup for 'u01' created successfully
SUCCESS: 2020-05-29 09:31:29: ...snapshot backup for 'root' created successfully
SUCCESS: 2020-05-29 09:31:29: LVM snapshots backup done successfully
[[email protected]~]#

Snapshots can be verified using the infosnap option

[[email protected]~]# /opt/odabr/odabr infosnap

│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│
 odabr - ODA node Backup Restore - Version: 2.0.1-55
 Copyright Oracle, Inc. 2013, 2020
 --------------------------------------------------------
 Author: Ruggero Citton 
 RAC Pack, Cloud Innovation and Solution Engineering Team
│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│


LVM snap name         Status                COW Size              Data%
-------------         ----------            ----------            ------
root_snap             active                20.00 GiB             0.02%
opt_snap              active                40.00 GiB             0.21%
u01_snap              active                80.00 GiB             0.02%


You have new mail in /var/spool/mail/root
[[email protected]~]#

We can now update the repository with the server software using the patch 31010832

For the first file

[[email protected]]# /opt/oracle/dcs/bin/odacli update-repository -f /u01/oda_patch_mdi/19.6/oda-sm-19.6.0.0.0-200420-server1of4.zip

{
  "jobId" : "e880e05e-e65a-44ea-897e-2ff376b28066",
  "status" : "Created",
  "message" : "/u01/oda_patch_mdi/19.6/oda-sm-19.6.0.0.0-200420-server1of4.zip",
  "reports" : [ ],
  "createTimestamp" : "May 29, 2020 10:11:11 AM CEST",
  "resourceList" : [ ],
  "description" : "Repository Update",
  "updatedTime" : "May 29, 2020 10:11:11 AM CEST"
}
[[email protected]~]#

Check the job status with describe-job

[[email protected]]# /opt/oracle/dcs/bin/odacli describe-job -i "e880e05e-e65a-44ea-897e-2ff376b28066"

Job details
----------------------------------------------------------------
                     ID:  e880e05e-e65a-44ea-897e-2ff376b28066
            Description:  Repository Update
                 Status:  Success
                Created:  May 29, 2020 10:11:11 AM CEST
                Message:  /u01/oda_patch_mdi/19.6/oda-sm-19.6.0.0.0-200420-server1of4.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

[[email protected]]#

Do this for all 3 other files. All jobs must return success.

[[email protected]]# /opt/oracle/dcs/bin/odacli update-repository -f /u01/oda_patch_mdi/19.6/oda-sm-19.6.0.0.0-200420-server2of4.zip
[[email protected]]# /opt/oracle/dcs/bin/odacli update-repository -f /u01/oda_patch_mdi/19.6/oda-sm-19.6.0.0.0-200420-server3of4.zip
[[email protected]]# /opt/oracle/dcs/bin/odacli update-repository -f /u01/oda_patch_mdi/19.6/oda-sm-19.6.0.0.0-200420-server4of4.zip

Now let’s update the DCS agent

[[email protected]]# /opt/oracle/dcs/bin/odacli update-dcsagent -v 19.6.0.0.0
{
  "jobId" : "2a2de2bc-d0c4-466a-91a8-818089d18867",
  "status" : "Created",
  "message" : "Dcs agent will be restarted after the update. Please wait for 2-3 mins before executing the other commands",
  "reports" : [ ],
  "createTimestamp" : "May 29, 2020 10:23:38 AM CEST",
  "resourceList" : [ ],
  "description" : "DcsAgent patching",
  "updatedTime" : "May 29, 2020 10:23:38 AM CEST"
}
[[email protected]]#

Confirm that the job returns success

[[email protected]]# /opt/oracle/dcs/bin/odacli describe-job -i "2a2de2bc-d0c4-466a-91a8-818089d18867"

Job details
----------------------------------------------------------------
                     ID:  2a2de2bc-d0c4-466a-91a8-818089d18867
            Description:  DcsAgent patching
                 Status:  Success
                Created:  May 29, 2020 10:23:38 AM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
dcs-agent upgrade  to version 19.6.0.0.0 May 29, 2020 10:23:38 AM CEST       May 29, 2020 10:25:17 AM CEST       Success
Update System version                    May 29, 2020 10:25:18 AM CEST       May 29, 2020 10:25:18 AM CEST       Success

[[email protected]]#

Before upgrading the OS, we must generate a prepatch report. This will indicate eventual issues that may cause the patching to fail.

[[email protected]]# /opt/oracle/dcs/bin/odacli create-prepatchreport -v 19.6.0.0.0 -os

Job details
----------------------------------------------------------------
                     ID:  d0a038e3-651d-478f-87a8-6cb1e125e437
            Description:  Patch pre-checks for [OS]
                 Status:  Created
                Created:  May 29, 2020 10:29:55 AM CEST
                Message:  Use 'odacli describe-prepatchreport -i d0a038e3-651d-478f-87a8-6cb1e125e437' to check details of results

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

[[email protected]]#

We can monitor the status of the job with the describe-job command. As we can see we have failure

[[email protected]]# /opt/oracle/dcs/bin/odacli describe-job -i d0a038e3-651d-478f-87a8-6cb1e125e437

Job details
----------------------------------------------------------------
                     ID:  d0a038e3-651d-478f-87a8-6cb1e125e437
            Description:  Patch pre-checks for [OS]
                 Status:  Failure
                Created:  May 29, 2020 10:29:55 AM CEST
                Message:  DCS-10001:Internal error encountered: One or more pre-checks failed. Run describe-prepatchreport for more details.

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
task:TaskZLockWrapper_132                May 29, 2020 10:30:01 AM CEST       May 29, 2020 10:40:02 AM CEST       Failure
Run patching pre-checks                  May 29, 2020 10:30:01 AM CEST       May 29, 2020 10:40:02 AM CEST       Success
Check pre-check status                   May 29, 2020 10:40:02 AM CEST       May 29, 2020 10:40:02 AM CEST       Failure

[[email protected]]#

To have more info about these errors, we use the describe-prepatchreport

[[email protected] 19.6]# /opt/oracle/dcs/bin/odacli describe-prepatchreport  -i d0a038e3-651d-478f-87a8-6cb1e125e437
 
Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  d0a038e3-651d-478f-87a8-6cb1e125e437
            Description:  Patch pre-checks for [OS]
                 Status:  FAILED
                Created:  May 29, 2020 10:29:55 AM CEST
                 Result:  One or more pre-checks failed for [OS]
 
Node Name
---------------
server-oda
 
Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.6.0.0.0.
Is patch location available     Success   Patch location is available.
Validate if ODABR is installed  Success   Validated ODABR is installed
Validate if ODABR snapshots     Failed    ODABR snapshots are seen on node:
exist                                     server-oda.
Validate LVM free space         Failed    Insufficient space to create LVM
                                          snapshots on node: server-oda. Expected
                                          free space (GB): 190, available space
                                        (GB): 22.
Space checks for OS upgrade     Success   Validated space checks.
Install OS upgrade software     Success   Extracted OS upgrade patches into
                                          /root/oda-upgrade. Do not remove this
                                          directory untill OS upgrade completes.
Verify OS upgrade by running    Success   Results stored in:
preupgrade checks                         '/root/preupgrade-results/
                                          preupg_results-200529103957.tar.gz' .
                                          Read complete report file
                                          '/root/preupgrade/result.html' before
                                          attempting OS upgrade.
Validate custom rpms installed  Failed    Custom RPMs installed. Please check
                                          files
                                          /root/oda-upgrade/
                                          rpms-added-from-ThirdParty and/or
                                          /root/oda-upgrade/
                                          rpms-added-from-Oracle.
Scheduled jobs check            Success    Scheduled jobs found. Disable
                                          scheduled jobs before attempting OS
                                          upgrade.
 
 
[[email protected] 19.6]#

As we can see the prepatch reports two errors
One related to odabr because I have already taken a backup for my ODA
A second error related to custom rpms installed.

For the error related to odabr as it is mentioned in the documentation

If snapshots are already present on the system when odacli create-prepatchreport is run, this precheck fails, because ODACLI expects to create these snapshots itself. If the user created snapshots or the operating system upgrade was retried (due to a failure) after it had already created the snapshots, this precheck will fail. Note that if snapshots already exist, odacli update-server –c OS still continues with the upgrade.

I decided to ignore the error and to run later the patch with option –force

But be sure before continuing to check these following two files and to remove all mentioned rpm packages.

/root/oda-upgrade/rpms-added-from-Oracle
/root/oda-upgrade/ rpms-added-from-ThirdParty

Just another thing os to verify if you have enough free space. You can if needed cleanup your old repository

[[email protected] 19.6]# odacli cleanup-patchrepo -comp GI,DB -v 18.8.0.0.0

In my case below is the status of my /, /u01 and /opt filesystems.

[[email protected]~]# df -h / /u01 /opt
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroupSys-LogVolRoot
                       30G   11G   18G  39% /
/dev/mapper/VolGroupSys-LogVolU01
                      148G   96G   45G  69% /u01
/dev/mapper/VolGroupSys-LogVolOpt
                       63G   40G   21G  66% /opt
[[email protected]~]#

It’s time now to start the patching process by upgrading the server to Linux 7. I did the following command from the ILOM console as described in the documentation.

With putty connect to your ILOM IP with root. Once connect to the ILOM just type start /SP/console and log in as root on your server.

-> start /SP/console
Are you sure you want to start /SP/console (y/n)? y

Serial console started.  To stop, type ESC (


Oracle Linux Server release 6.10
Kernel 4.1.12-124.33.4.el6uek.x86_64 on an x86_64

oda-serverlogin: root
Password:
Last login: Fri May 29 08:28:34 from 10.14.211.131
[[email protected]~]#

And with root from the ILOM console run the OS upgrade (Just remember in my case I decided to use the –force)

[[email protected]~]# odacli update-server -v 19.6.0.0.0 -c os --local --force
Verifying OS upgrade
Current OS base version: 6 is lessthan target OS base version: 7
OS needs to upgrade to 7.7
****************************************************************************
*  Depending on the hardware platform, the upgrade operation, including    *
*  node reboot, may take 30-60 minutes to complete. Individual steps in    *
*  the operation may not show progress messages for a while. Do not abort  *
*  upgrade using ctrl-c or by rebooting the system.                        *
****************************************************************************

run: cmd= '[/usr/bin/expect, /root/upgradeos.exp]'
output redirected to /root/odaUpgrade2020-05-29_13-50-23.0388.log

Running pre-upgrade checks.

Running pre-upgrade checks.
........
Running assessment of the system
.........

This will take some time, but just wait. You may see the progression from the ILOM console

…
[  112.602728] upgrade[6905]: [505/779] (51%) installing libsmbclient-4.9.1-10.el7_7...
[  112.647750] upgrade[6905]: [506/779] (51%) installing mesa-libgbm-18.3.4-6.el7_7...
[  112.691582] upgrade[6905]: [507/779] (51%) installing device-mapper-event-1.02.158-2.0.1.el7_7.2...
[  112.719106] upgrade[6905]: Created symlink /etc/systemd/system/sockets.target.wants/dm-event.socket, pointing to /usr/lib/systemd/system/dm-event.socket.
[  112.736417] upgrade[6905]: Running in chroot, ignoring request.
[  112.745205] upgrade[6905]: [508/779] (51%) installing usbredir-0.7.1-3.el7...
[  112.780151] upgrade[6905]: [509/779] (51%) installing uptrack-offline-1.2.62.offline-0.el7...
[  113.001966] upgrade[6905]: [510/779] (51%) installing dconf-0.28.0-4.el7...
[  113.129938] upgrade[6905]: [511/779] (51%) installing iscsi-initiator-utils-iscsiuio-6.2.0.874-11.0.1.el7...
[  113.178038] upgrade[6905]: [512/779] (51%) installing iscsi-initiator-utils-6.2.0.874-11.0.1.el7...
[  113.362242] upgrade[6905]: [513/779] (52%) installing unbound-libs-1.6.6-1.el7...

At the end of the process the server will reboot. And if everything is fine you should have an Linux 7.7 now

[[email protected]~]# cat /etc/oracle-release
Oracle Linux Server release 7.7
[[email protected]~]#

After the operating system upgrade is completed successfully, run the post upgrade checks:

[[email protected]~]# /opt/oracle/dcs/bin/odacli update-server-postcheck -v 19.6.0.0.0

Upgrade post-check report
-------------------------

Node Name
---------------
server-oda

Comp  Pre-Check                      Status   Comments
----- ------------------------------ -------- --------------------------------------
OS    OS upgrade check               SUCCESS  OS has been upgraded to OL7
GI    GI upgrade check               INFO     GI home needs to update to 19.6.0.0.200114
GI    GI status check                SUCCESS  Clusterware is running on the node
OS    ODABR snapshot                 WARNING  ODABR snapshot found. Run 'odabr delsnap' to delete.
RPM   Extra RPM check                SUCCESS  No extra RPMs found when OS was at OL6
[[email protected]~]#

As specified let’s remove the snapshots we took with odabr

[[email protected]~]# /opt/odabr/odabr delsnap
INFO: 2020-05-29 14:22:31: Please check the logfile '/opt/odabr/out/log/odabr_60717.log' for more details

INFO: 2020-05-29 14:22:31: Removing LVM snapshots
INFO: 2020-05-29 14:22:31: ...removing LVM snapshot for 'opt'
SUCCESS: 2020-05-29 14:22:32: ...snapshot for 'opt' removed successfully
INFO: 2020-05-29 14:22:32: ...removing LVM snapshot for 'u01'
SUCCESS: 2020-05-29 14:22:32: ...snapshot for 'u01' removed successfully
INFO: 2020-05-29 14:22:32: ...removing LVM snapshot for 'root'
SUCCESS: 2020-05-29 14:22:32: ...snapshot for 'root' removed successfully
SUCCESS: 2020-05-29 14:22:32: Remove LVM snapshots done successfully

Running the posthecks again

[[email protected]~]# /opt/oracle/dcs/bin/odacli update-server-postcheck -v 19.6.0.0.0

Upgrade post-check report
-------------------------

Node Name
---------------
server-oda

Comp  Pre-Check                      Status   Comments
----- ------------------------------ -------- --------------------------------------
OS    OS upgrade check               SUCCESS  OS has been upgraded to OL7
GI    GI upgrade check               INFO     GI home needs to update to 19.6.0.0.200114
GI    GI status check                SUCCESS  Clusterware is running on the node
OS    ODABR snapshot                 SUCCESS  No ODABR snapshots found
RPM   Extra RPM check                SUCCESS  No extra RPMs found when OS was at OL6
[[email protected]~]#

After the OS upgrade now let’s upgrade the remaining components
We start by updating the DSC agent

[[email protected]~]# /opt/oracle/dcs/bin/odacli update-dcsadmin -v 19.6.0.0.0
{
  "jobId" : "5bdb8d99-d3ae-421c-a05d-60b9ece94021",
  "status" : "Created",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : "May 29, 2020 14:26:32 PM CEST",
  "resourceList" : [ ],
  "description" : "DcsAdmin patching",
  "updatedTime" : "May 29, 2020 14:26:32 PM CEST"
}
[[email protected]~]#

We validate that the Job returns success

[[email protected]~]# /opt/oracle/dcs/bin/odacli describe-job -i "5bdb8d99-d3ae-421c-a05d-60b9ece94021"

Job details
----------------------------------------------------------------
                     ID:  5bdb8d99-d3ae-421c-a05d-60b9ece94021
            Description:  DcsAdmin patching
                 Status:  Success
                Created:  May 29, 2020 2:26:32 PM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation                May 29, 2020 2:26:32 PM CEST        May 29, 2020 2:26:32 PM CEST        Success
dcsadmin upgrade                         May 29, 2020 2:26:32 PM CEST        May 29, 2020 2:26:32 PM CEST        Success
Update System version                    May 29, 2020 2:26:32 PM CEST        May 29, 2020 2:26:32 PM CEST        Success

[[email protected]~]#

We update the DCS components

[[email protected]~]# /opt/oracle/dcs/bin/odacli update-dcscomponents -v 19.6.0.0.0
{
  "jobId" : "4af0626e-d260-4f93-8948-2f241f7b2c48",
  "status" : "Success",
  "message" : null,
  "reports" : null,
  "createTimestamp" : "May 29, 2020 14:27:52 PM CEST",
  "description" : "Job completed and is not part of Agent job list",
  "updatedTime" : "May 29, 2020 14:27:52 PM CEST"
}
[[email protected]~]#

And then we update the server

[[email protected]~]# /opt/oracle/dcs/bin/odacli update-server -v 19.6.0.0.0
{
  "jobId" : "91f7b206-b097-4ba2-bae7-42a6b3bb0ba6",
  "status" : "Created",
  "message" : "Success of server update will trigger reboot of the node after 4-5 minutes. Please wait until the node reboots.",
  "reports" : [ ],
  "createTimestamp" : "May 29, 2020 14:31:25 PM CEST",
  "resourceList" : [ ],
  "description" : "Server Patching",
  "updatedTime" : "May 29, 2020 14:31:25 PM CEST"
}
[[email protected]~]#

This will lasts about 45 minutes. Validate that all is fine

[[email protected]~]# /opt/oracle/dcs/bin/odacli describe-job -i "91f7b206-b097-4ba2-bae7-42a6b3bb0ba6"

Job details
----------------------------------------------------------------
                     ID:  91f7b206-b097-4ba2-bae7-42a6b3bb0ba6
            Description:  Server Patching
                 Status:  Success
                Created:  May 29, 2020 2:31:25 PM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation                May 29, 2020 2:31:33 PM CEST        May 29, 2020 2:31:33 PM CEST        Success
dcs-controller upgrade                   May 29, 2020 2:31:33 PM CEST        May 29, 2020 2:31:33 PM CEST        Success
Creating repositories using yum          May 29, 2020 2:31:33 PM CEST        May 29, 2020 2:31:34 PM CEST        Success
Applying HMP Patches                     May 29, 2020 2:31:34 PM CEST        May 29, 2020 2:31:34 PM CEST        Success
Patch location validation                May 29, 2020 2:31:34 PM CEST        May 29, 2020 2:31:34 PM CEST        Success
oda-hw-mgmt upgrade                      May 29, 2020 2:31:35 PM CEST        May 29, 2020 2:31:35 PM CEST        Success
OSS Patching                             May 29, 2020 2:31:35 PM CEST        May 29, 2020 2:31:35 PM CEST        Success
Applying Firmware Disk Patches           May 29, 2020 2:31:35 PM CEST        May 29, 2020 2:31:40 PM CEST        Success
Applying Firmware Expander Patches       May 29, 2020 2:31:40 PM CEST        May 29, 2020 2:31:44 PM CEST        Success
Applying Firmware Controller Patches     May 29, 2020 2:31:44 PM CEST        May 29, 2020 2:31:48 PM CEST        Success
Checking Ilom patch Version              May 29, 2020 2:31:50 PM CEST        May 29, 2020 2:31:52 PM CEST        Success
Patch location validation                May 29, 2020 2:31:52 PM CEST        May 29, 2020 2:31:53 PM CEST        Success
Save password in Wallet                  May 29, 2020 2:31:54 PM CEST        May 29, 2020 2:31:54 PM CEST        Success
Apply Ilom patch                         May 29, 2020 2:31:54 PM CEST        May 29, 2020 2:39:55 PM CEST        Success
Copying Flash Bios to Temp location      May 29, 2020 2:39:55 PM CEST        May 29, 2020 2:39:55 PM CEST        Success
Starting the clusterware                 May 29, 2020 2:39:55 PM CEST        May 29, 2020 2:41:52 PM CEST        Success
Creating GI home directories             May 29, 2020 2:41:53 PM CEST        May 29, 2020 2:41:53 PM CEST        Success
Cloning Gi home                          May 29, 2020 2:41:53 PM CEST        May 29, 2020 2:44:16 PM CEST        Success
Configuring GI                           May 29, 2020 2:44:16 PM CEST        May 29, 2020 2:46:25 PM CEST        Success
Running GI upgrade root scripts          May 29, 2020 2:46:25 PM CEST        May 29, 2020 3:03:15 PM CEST        Success
Resetting DG compatibility               May 29, 2020 3:03:15 PM CEST        May 29, 2020 3:03:33 PM CEST        Success
Running GI config assistants             May 29, 2020 3:03:33 PM CEST        May 29, 2020 3:04:18 PM CEST        Success
restart oakd                             May 29, 2020 3:04:28 PM CEST        May 29, 2020 3:04:39 PM CEST        Success
Updating GiHome version                  May 29, 2020 3:04:39 PM CEST        May 29, 2020 3:04:45 PM CEST        Success
Update System version                    May 29, 2020 3:04:54 PM CEST        May 29, 2020 3:04:54 PM CEST        Success
preRebootNode Actions                    May 29, 2020 3:04:54 PM CEST        May 29, 2020 3:05:40 PM CEST        Success
Reboot Ilom                              May 29, 2020 3:05:40 PM CEST        May 29, 2020 3:05:40 PM CEST        Success

[[email protected]~]#

And we can verify that the components were upgraded

[[email protected]~]# /opt/oracle/dcs/bin/odacli describe-component
System Version
---------------
19.6.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.6.0.0.0            up-to-date
GI                                        19.6.0.0.200114       up-to-date
DB                                        11.2.0.4.190115       11.2.0.4.200114
DCSAGENT                                  19.6.0.0.0            up-to-date
ILOM                                      4.0.4.52.r133103      up-to-date
BIOS                                      41060700              up-to-date
OS                                        7.7                   up-to-date
FIRMWARECONTROLLER                        QDV1RF30              up-to-date
FIRMWAREDISK                              0121                  up-to-date
HMP                                       2.4.5.0.1             up-to-date

[[email protected]~]#

After the server we have to update the storage

[[email protected]~]# /opt/oracle/dcs/bin/odacli update-storage -v 19.6.0.0.0
{
  "jobId" : "c4b365ff-bd8c-4bca-b53d-d9d9bda90548",
  "status" : "Created",
  "message" : "Success of Storage Update may trigger reboot of node after 4-5 minutes. Please wait till node restart",
  "reports" : [ ],
  "createTimestamp" : "May 29, 2020 15:16:17 PM CEST",
  "resourceList" : [ ],
  "description" : "Storage Firmware Patching",
  "updatedTime" : "May 29, 2020 15:16:17 PM CEST"
}
[[email protected]~]#

We can verify that the job was successful

[[email protected]~]# /opt/oracle/dcs/bin/odacli describe-job -i "c4b365ff-bd8c-4bca-b53d-d9d9bda90548"

Job details
----------------------------------------------------------------
                     ID:  c4b365ff-bd8c-4bca-b53d-d9d9bda90548
            Description:  Storage Firmware Patching
                 Status:  Success
                Created:  May 29, 2020 3:16:17 PM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Applying Firmware Disk Patches           May 29, 2020 3:16:18 PM CEST        May 29, 2020 3:16:21 PM CEST        Success
Applying Firmware Controller Patches     May 29, 2020 3:16:21 PM CEST        May 29, 2020 3:16:26 PM CEST        Success
preRebootNode Actions                    May 29, 2020 3:16:26 PM CEST        May 29, 2020 3:16:26 PM CEST        Success
Reboot Ilom                              May 29, 2020 3:16:26 PM CEST        May 29, 2020 3:16:26 PM CEST        Success

[[email protected]~]#

I did not patch the existing database home because I just want to keep the actual version for my existing databases.

To be able to create new 19c databases we have to update the repository with corresponding rdbms clone.
After unzipping the archive

[email protected]:/u01/oda_patch_mdi/19.6/ [ORCL] unzip p30403662_196000_Linux-x86-64.zip
Archive:  p30403662_196000_Linux-x86-64.zip
 extracting: odacli-dcs-19.6.0.0.0-200326-DB-19.6.0.0.zip

I just have to update the repository with the db clone

[[email protected]]# /opt/oracle/dcs/bin/odacli update-repository -f /u01/oda_patch_mdi/19.6/odacli-dcs-19.6.0.0.0-200326-DB-19.6.0.0.zip
{
  "jobId" : "e8e58467-460f-4b07-86db-d4b71f1e5884",
  "status" : "Created",
  "message" : "/u01/oda_patch_mdi/19.6/odacli-dcs-19.6.0.0.0-200326-DB-19.6.0.0.zip",
  "reports" : [ ],
  "createTimestamp" : "May 29, 2020 15:23:37 PM CEST",
  "resourceList" : [ ],
  "description" : "Repository Update",
  "updatedTime" : "May 29, 2020 15:23:37 PM CEST"
}
[[email protected]]#

Then I check that the job was fine

[[email protected]]# /opt/oracle/dcs/bin/odacli describe-job -i "e8e58467-460f-4b07-86db-d4b71f1e5884"

Job details
----------------------------------------------------------------
                     ID:  e8e58467-460f-4b07-86db-d4b71f1e5884
            Description:  Repository Update
                 Status:  Success
                Created:  May 29, 2020 3:23:37 PM CEST
                Message:  /u01/oda_patch_mdi/19.6/odacli-dcs-19.6.0.0.0-200326-DB-19.6.0.0.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

[[email protected]]#

Encountered ACFS Issue
During the patch I did not have any issue. All was fine. But at the end of the patch my existing databases did not come up because of an acfs issue.
Indeed, the ASM instance was up but the ASM proxy instance APX was instable

[[email protected]~]$ srvctl start  asm -proxy -node oda-server
PRCR-1013 : Failed to start resource ora.proxy_advm
PRCR-1064 : Failed to start resource ora.proxy_advm on node oda-server
CRS-5017: The resource action "ora.proxy_advm start" encountered the following error:
ORA-01092: ORACLE instance terminated. Disconnection forced
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/oda-server/crs/trace/crsd_oraagent_grid.trc".

Looking on the trace file

[[email protected]]# grep -i ORA- /u01/app/grid/diag/crs/oda-server/crs/trace/crsd_oraagent_grid.trc
2020-05-29 16:51:18.216 :CLSDYNAM:916408064: [ora.proxy_advm]{1:38257:2942} [start] ORA-03113: end-of-file on communication channel
2020-05-29 16:51:18.216 :CLSDYNAM:916408064: [ora.proxy_advm]{1:38257:2942} [start] InstAgent::startInstance 250 ORA-3113retryCount:0 m_instanceType:2 m_lastOCIError:3113
2020-05-29 16:51:39.641 :CLSDYNAM:916408064: [ora.proxy_advm]{1:38257:2942} [start] ORA-03113: end-of-file on communication channel
2020-05-29 16:51:39.641 :CLSDYNAM:916408064: [ora.proxy_advm]{1:38257:2942} [start] InstAgent::startInstance 250 ORA-3113retryCount:1 m_instanceType:2 m_lastOCIError:3113
2020-05-29 16:52:01.078 :CLSDYNAM:916408064: [ora.proxy_advm]{1:38257:2942} [start] ORA-03113: end-of-file on communication channel
2020-05-29 16:52:01.078 :CLSDYNAM:916408064: [ora.proxy_advm]{1:38257:2942} [start] InstAgent::startInstance 250 ORA-3113retryCount:2 m_instanceType:2 m_lastOCIError:3113
2020-05-29 16:52:01.098 :CLSDYNAM:916408064: [ora.proxy_advm]{1:38257:2942} [start] ORA-01092: ORACLE instance terminated. Disconnection forced
2020-05-29 16:52:01.099 :CLSDYNAM:916408064: [ora.proxy_advm]{1:38257:2942} [start] InstAgent::startInstance 380 throw excp what:ORA-01092: ORACLE instance terminated. Disconnection forced
2020-05-29 16:52:01.099 :CLSDYNAM:916408064: [ora.proxy_advm]{1:38257:2942} [start] ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-01092: ORACLE instance terminated. Disconnection forced
[[email protected]]#

For some reason the acfs module was not loaded. My databases was Oracle 11.2.0.4 and were using acfs.

[[email protected]~]# /sbin/lsmod | grep oracle
oracleoks             724992  0
oracleafd             229376  0

So I first stop my crs

[[email protected]~]#  crsctl stop crs

And then I reinstall the acfs module with following acfsroot command

[[email protected]~]# which acfsroot
/u01/app/19.0.0.0/grid/bin/acfsroot

[[email protected]~]# /u01/app/19.0.0.0/grid/bin/acfsroot install
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9294: updating file /etc/sysconfig/oracledrivers.conf
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9294: updating file /etc/sysconfig/oracledrivers.conf
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9154: Loading 'oracleoks.ko' driver.
ACFS-9154: Loading 'oracleadvm.ko' driver.
ACFS-9154: Loading 'oracleacfs.ko' driver.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9309: ADVM/ACFS installation correctness verified.
[[email protected]~]#

Running again the lsmod command, I see that things are better

[[email protected]~]# /sbin/lsmod | grep oracle
oracleacfs           5828608  0
oracleadvm           1204224  0
oracleoks             724992  2 oracleadvm,oracleacfs
oracleafd             229376  0
[[email protected]~]#

After I reboot my server and all databases and the ASM Proxy Instance become up and running. I just wanted to mention this issue I had and hope this will help.

Conclusion

Some recommendations
Remove all rpm packages that were manually installed
Verify that you have enough space on / , /u01 and /opt filesystems. /opt and /u01 can be increased online
Launch the OS upgrade from the ILOM console
I hope this blog will help.

2 Comments

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Mouhamadou Diaw
Mouhamadou Diaw

Consultant