By Mouhamadou Diaw
The ODA software 19.6 is released and people are starting to patch.
A direct patch to version 19.6 is possible from version 18.8.
Before patching your deployment to Oracle Database Appliance release 19.6, you must upgrade the operating system to Oracle Linux 7.
In this blog I am describing the steps I follow when patching an ODA from 18.8 to 19.6. I am using an ODA X7-2 (one node)
The first step is of course to download the required patch. You will need to copy following patch to your ODA.
Patch 31010832 : ORACLE DATABASE APPLIANCE 19.6.0.0.0 SERVER PATCH FOR ALL ODACLI/DCS STACK
Patch 30403662 : ORACLE DATABASE APPLIANCE RDBMS CLONE FOR ODACLI/DCS STACK
The first patch 31010832 will be used to upgrade the OS to Linux 7 and to patch you server to Appliance 19.6
The patch 30403662 will install the Oracle 19c clone database
The first patch contains 4 files
1
2
3
4
|
p31010832_196000_Linux-x86-64_1of4.zip p31010832_196000_Linux-x86-64_2of4.zip p31010832_196000_Linux-x86-64_3of4.zip p31010832_196000_Linux-x86-64_4of4.zip |
Just use the unzip command to decompress
1
2
3
4
|
oracle@server-oda: /u01/oda_patch_mdi/19 .6/ [ORCL] unzip p31010832_196000_Linux-x86-64_1of4.zip oracle@server-oda: /u01/oda_patch_mdi/19 .6/ [ORCL] unzip p31010832_196000_Linux-x86-64_2of4.zip oracle@server-oda: /u01/oda_patch_mdi/19 .6/ [ORCL] unzip p31010832_196000_Linux-x86-64_3of4.zip oracle@server-oda: /u01/oda_patch_mdi/19 .6/ [ORCL] unzip p31010832_196000_Linux-x86-64_4of4.zip |
Once done we have following files that we will use to update the repository
1
2
3
4
|
oda-sm-19.6.0.0.0-200420-server1of4.zip oda-sm-19.6.0.0.0-200420-server2of4.zip oda-sm-19.6.0.0.0-200420-server3of4.zip oda-sm-19.6.0.0.0-200420-server4of4.zip |
After you will have to list your scheduled jobs with the list-schedules option
1
2
3
4
5
6
7
8
9
|
[root@oda-serveru01] # odacli list-schedules ID Name Description CronExpression Disabled ---------------------------------------- ------------------------- -------------------------------------------------- ------------------------------ -------- 6d9cd445-8890-4bd6-a713-f6eb8fce35d0 metastore maintenance internal metastore maintenance 0 0 0 1 /1 * ? * true 113ea801-636c-45d9-ad70-448054d825d5 AgentState metastore cleanup internal agentstateentry metastore maintenance 0 0 0 1 /1 * ? * false e1780bf3-2467-4a89-9298-857fed7fa101 bom maintenance bom reports generation 0 0 1 ? * SUN * false 2f39ed40-6fc9-4547-b9f1-11889c5c7df9 Big File Upload Cleanup clean up expired big file uploads. 0 0 1 ? * SUN * false a624a5c4-4801-444b-9b48-8be99f9f9e48 feature_tracking_job Feature tracking job 0 0 20 ? * WED * false |
And disable all enabled jobs (column disabled to false). This done by using the command update-server with -d for disable and -i for the job_id
1
2
3
4
5
6
7
8
9
|
[root@oda-server~] # odacli update-schedule -d -i 113ea801-636c-45d9-ad70-448054d825d5 Update job schedule success [root@oda-server~] # odacli update-schedule -d -i e1780bf3-2467-4a89-9298-857fed7fa101 Update job schedule success [root@oda-server~] # odacli update-schedule -d -i 2f39ed40-6fc9-4547-b9f1-11889c5c7df9 Update job schedule success [root@oda-server~] # odacli update-schedule -d -i a624a5c4-4801-444b-9b48-8be99f9f9e48 Update job schedule success [root@oda-server~] # |
When listing the jobs again, the column disabled must return true for all jobs
odabr is required. for the patching to 19.6. odabr is a tool to backup and recover an ODA. When running the precheck. the result will return failed if odabr is not installed. Please you can consult following document for downloading and using odabr : ODA (Oracle Database Appliance): ODABR a System Backup/Restore Utility (Doc ID 2466177.1)
By default odabr requires 190G free space in the LVM which can be not the case sometimes. In such case odabr should be run using some specific options. In my case I used the command below to take a backup of ODA.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
|
[root@oda-server~] # /opt/odabr/odabr backup -snap -osize 40 -rsize 20 -usize 80 INFO: 2020-05-29 09:31:15: Please check the logfile '/opt/odabr/out/log/odabr_81097.log' for more details │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│ odabr - ODA node Backup Restore - Version: 2.0.1-55 Copyright Oracle, Inc. 2013, 2020 -------------------------------------------------------- Author: Ruggero Citton RAC Pack, Cloud Innovation and Solution Engineering Team │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│ INFO: 2020-05-29 09:31:15: Checking superuser INFO: 2020-05-29 09:31:15: Checking Bare Metal INFO: 2020-05-29 09:31:15: Removing existing LVM snapshots WARNING: 2020-05-29 09:31:15: LVM snapshot for 'opt' does not exist WARNING: 2020-05-29 09:31:15: LVM snapshot for 'u01' does not exist WARNING: 2020-05-29 09:31:16: LVM snapshot for 'root' does not exist INFO: 2020-05-29 09:31:16: Checking LVM size INFO: 2020-05-29 09:31:16: Boot device backup INFO: 2020-05-29 09:31:16: ...getting boot device INFO: 2020-05-29 09:31:16: ...making boot device backup INFO: 2020-05-29 09:31:20: ...boot device backup saved as '/opt/odabr/out/hbi/boot.img' INFO: 2020-05-29 09:31:21: ...boot device backup check passed INFO: 2020-05-29 09:31:21: Getting EFI device INFO: 2020-05-29 09:31:21: ...making efi device backup INFO: 2020-05-29 09:31:24: ...EFI device backup saved as '/opt/odabr/out/hbi/efi.img' INFO: 2020-05-29 09:31:24: ...EFI device backup check passed INFO: 2020-05-29 09:31:24: OCR backup INFO: 2020-05-29 09:31:26: ...ocr backup saved as '/opt/odabr/out/hbi/ocrbackup_81097.bck' INFO: 2020-05-29 09:31:26: Making LVM snapshot backup SUCCESS: 2020-05-29 09:31:28: ...snapshot backup for 'opt' created successfully SUCCESS: 2020-05-29 09:31:29: ...snapshot backup for 'u01' created successfully SUCCESS: 2020-05-29 09:31:29: ...snapshot backup for 'root' created successfully SUCCESS: 2020-05-29 09:31:29: LVM snapshots backup done successfully [root@oda-server~] # |
Snapshots can be verified using the infosnap option
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
[root@oda-server~] # /opt/odabr/odabr infosnap │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│ odabr - ODA node Backup Restore - Version: 2.0.1-55 Copyright Oracle, Inc. 2013, 2020 -------------------------------------------------------- Author: Ruggero Citton RAC Pack, Cloud Innovation and Solution Engineering Team │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│ LVM snap name Status COW Size Data% ------------- ---------- ---------- ------ root_snap active 20.00 GiB 0.02% opt_snap active 40.00 GiB 0.21% u01_snap active 80.00 GiB 0.02% You have new mail in /var/spool/mail/root [root@oda-server~] # |
We can now update the repository with the server software using the patch 31010832
For the first file
1
2
3
4
5
6
7
8
9
10
11
12
13
|
[[email protected]] # /opt/oracle/dcs/bin/odacli update-repository -f /u01/oda_patch_mdi/19.6/oda-sm-19.6.0.0.0-200420-server1of4.zip { "jobId" : "e880e05e-e65a-44ea-897e-2ff376b28066" , "status" : "Created" , "message" : "/u01/oda_patch_mdi/19.6/oda-sm-19.6.0.0.0-200420-server1of4.zip" , "reports" : [ ], "createTimestamp" : "May 29, 2020 10:11:11 AM CEST" , "resourceList" : [ ], "description" : "Repository Update" , "updatedTime" : "May 29, 2020 10:11:11 AM CEST" } [root@oda-server~] # |
Check the job status with describe-job
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
[[email protected]] # /opt/oracle/dcs/bin/odacli describe-job -i "e880e05e-e65a-44ea-897e-2ff376b28066" Job details ---------------------------------------------------------------- ID: e880e05e-e65a-44ea-897e-2ff376b28066 Description: Repository Update Status: Success Created: May 29, 2020 10:11:11 AM CEST Message: /u01/oda_patch_mdi/19 .6 /oda-sm-19 .6.0.0.0-200420-server1of4.zip Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- |
Do this for all 3 other files. All jobs must return success.
1
2
3
|
[[email protected]] # /opt/oracle/dcs/bin/odacli update-repository -f /u01/oda_patch_mdi/19.6/oda-sm-19.6.0.0.0-200420-server2of4.zip [[email protected]] # /opt/oracle/dcs/bin/odacli update-repository -f /u01/oda_patch_mdi/19.6/oda-sm-19.6.0.0.0-200420-server3of4.zip [[email protected]] # /opt/oracle/dcs/bin/odacli update-repository -f /u01/oda_patch_mdi/19.6/oda-sm-19.6.0.0.0-200420-server4of4.zip |
Now let’s update the DCS agent
1
2
3
4
5
6
7
8
9
10
11
12
|
[[email protected]] # /opt/oracle/dcs/bin/odacli update-dcsagent -v 19.6.0.0.0 { "jobId" : "2a2de2bc-d0c4-466a-91a8-818089d18867" , "status" : "Created" , "message" : "Dcs agent will be restarted after the update. Please wait for 2-3 mins before executing the other commands" , "reports" : [ ], "createTimestamp" : "May 29, 2020 10:23:38 AM CEST" , "resourceList" : [ ], "description" : "DcsAgent patching" , "updatedTime" : "May 29, 2020 10:23:38 AM CEST" } |
Confirm that the job returns success
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
[[email protected]] # /opt/oracle/dcs/bin/odacli describe-job -i "2a2de2bc-d0c4-466a-91a8-818089d18867" Job details ---------------------------------------------------------------- ID: 2a2de2bc-d0c4-466a-91a8-818089d18867 Description: DcsAgent patching Status: Success Created: May 29, 2020 10:23:38 AM CEST Message: Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- dcs-agent upgrade to version 19.6.0.0.0 May 29, 2020 10:23:38 AM CEST May 29, 2020 10:25:17 AM CEST Success Update System version May 29, 2020 10:25:18 AM CEST May 29, 2020 10:25:18 AM CEST Success |
Before upgrading the OS, we must generate a prepatch report. This will indicate eventual issues that may cause the patching to fail.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
[[email protected]] # /opt/oracle/dcs/bin/odacli create-prepatchreport -v 19.6.0.0.0 -os Job details ---------------------------------------------------------------- ID: d0a038e3-651d-478f-87a8-6cb1e125e437 Description: Patch pre-checks for [OS] Status: Created Created: May 29, 2020 10:29:55 AM CEST Message: Use 'odacli describe-prepatchreport -i d0a038e3-651d-478f-87a8-6cb1e125e437' to check details of results Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- |
We can monitor the status of the job with the describe-job command. As we can see we have failure
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
[[email protected]] # /opt/oracle/dcs/bin/odacli describe-job -i d0a038e3-651d-478f-87a8-6cb1e125e437 Job details ---------------------------------------------------------------- ID: d0a038e3-651d-478f-87a8-6cb1e125e437 Description: Patch pre-checks for [OS] Status: Failure Created: May 29, 2020 10:29:55 AM CEST Message: DCS-10001:Internal error encountered: One or more pre-checks failed. Run describe-prepatchreport for more details. Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- task:TaskZLockWrapper_132 May 29, 2020 10:30:01 AM CEST May 29, 2020 10:40:02 AM CEST Failure Run patching pre-checks May 29, 2020 10:30:01 AM CEST May 29, 2020 10:40:02 AM CEST Success Check pre-check status May 29, 2020 10:40:02 AM CEST May 29, 2020 10:40:02 AM CEST Failure |
To have more info about these errors, we use the describe-prepatchreport
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
|
[root@server-oda 19.6] # /opt/oracle/dcs/bin/odacli describe-prepatchreport -i d0a038e3-651d-478f-87a8-6cb1e125e437 Patch pre-check report ------------------------------------------------------------------------ Job ID: d0a038e3-651d-478f-87a8-6cb1e125e437 Description: Patch pre-checks for [OS] Status: FAILED Created: May 29, 2020 10:29:55 AM CEST Result: One or more pre-checks failed for [OS] Node Name --------------- server-oda Pre-Check Status Comments ------------------------------ -------- -------------------------------------- __OS__ Validate supported versions Success Validated minimum supported versions. Validate patching tag Success Validated patching tag: 19.6.0.0.0. Is patch location available Success Patch location is available. Validate if ODABR is installed Success Validated ODABR is installed Validate if ODABR snapshots Failed ODABR snapshots are seen on node: exist server-oda. Validate LVM free space Failed Insufficient space to create LVM snapshots on node: server-oda. Expected free space (GB): 190, available space (GB): 22. Space checks for OS upgrade Success Validated space checks. Install OS upgrade software Success Extracted OS upgrade patches into /root/oda-upgrade . Do not remove this directory untill OS upgrade completes. Verify OS upgrade by running Success Results stored in : preupgrade checks ' /root/preupgrade-results/ preupg_results-200529103957. tar .gz' . Read complete report file '/root/preupgrade/result.html' before attempting OS upgrade. Validate custom rpms installed Failed Custom RPMs installed. Please check files /root/oda-upgrade/ rpms-added-from-ThirdParty and /or /root/oda-upgrade/ rpms-added-from-Oracle. Scheduled jobs check Success Scheduled jobs found. Disable scheduled jobs before attempting OS upgrade. [root@server-oda 19.6] # |
As we can see the prepatch reports two errors
One related to odabr because I have already taken a backup for my ODA
A second error related to custom rpms installed.
For the error related to odabr as it is mentioned in the documentation
If snapshots are already present on the system when odacli create-prepatchreport is run, this precheck fails, because ODACLI expects to create these snapshots itself. If the user created snapshots or the operating system upgrade was retried (due to a failure) after it had already created the snapshots, this precheck will fail. Note that if snapshots already exist, odacli update-server –c OS still continues with the upgrade.
I decided to ignore the error and to run later the patch with option –force
But be sure before continuing to check these following two files and to remove all mentioned rpm packages.
1
2
|
/root/oda-upgrade/rpms-added-from-Oracle /root/oda-upgrade/ rpms-added-from-ThirdParty |
Just another thing os to verify if you have enough free space. You can if needed cleanup your old repository
1
|
[root@server-oda 19.6] # odacli cleanup-patchrepo -comp GI,DB -v 18.8.0.0.0 |
In my case below is the status of my /, /u01 and /opt filesystems.
1
2
3
4
5
6
7
8
9
|
[root@oda-server~] # df -h / /u01 /opt Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroupSys-LogVolRoot 30G 11G 18G 39% / /dev/mapper/VolGroupSys-LogVolU01 148G 96G 45G 69% /u01 /dev/mapper/VolGroupSys-LogVolOpt 63G 40G 21G 66% /opt [root@oda-server~] # |
It’s time now to start the patching process by upgrading the server to Linux 7. I did the following command from the ILOM console as described in the documentation.
With putty connect to your ILOM IP with root. Once connect to the ILOM just type start /SP/console and log in as root on your server.
1
2
3
4
5
6
7
8
9
10
11
12
13
|
-> start /SP/console Are you sure you want to start /SP/console (y /n )? y Serial console started. To stop, type ESC ( Oracle Linux Server release 6.10 Kernel 4.1.12-124.33.4.el6uek.x86_64 on an x86_64 oda-serverlogin: root Password: Last login: Fri May 29 08:28:34 from 10.14.211.131 [root@oda-server~] # |
And with root from the ILOM console run the OS upgrade (Just remember in my case I decided to use the –force)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
[root@oda-server~] # odacli update-server -v 19.6.0.0.0 -c os --local --force Verifying OS upgrade Current OS base version: 6 is lessthan target OS base version: 7 OS needs to upgrade to 7.7 **************************************************************************** * Depending on the hardware platform, the upgrade operation, including * * node reboot, may take 30-60 minutes to complete. Individual steps in * * the operation may not show progress messages for a while . Do not abort * * upgrade using ctrl-c or by rebooting the system. * **************************************************************************** run: cmd= '[/usr/bin/expect, /root/upgradeos.exp]' output redirected to /root/odaUpgrade2020-05-29_13-50-23 .0388.log Running pre-upgrade checks. Running pre-upgrade checks. ........ Running assessment of the system ......... |
This will take some time, but just wait. You may see the progression from the ILOM console
1
2
3
4
5
6
7
8
9
10
11
12
|
… [ 112.602728] upgrade[6905]: [505 /779 ] (51%) installing libsmbclient-4.9.1-10.el7_7... [ 112.647750] upgrade[6905]: [506 /779 ] (51%) installing mesa-libgbm-18.3.4-6.el7_7... [ 112.691582] upgrade[6905]: [507 /779 ] (51%) installing device-mapper-event-1.02.158-2.0.1.el7_7.2... [ 112.719106] upgrade[6905]: Created symlink /etc/systemd/system/sockets .target.wants /dm-event .socket, pointing to /usr/lib/systemd/system/dm-event .socket. [ 112.736417] upgrade[6905]: Running in chroot, ignoring request. [ 112.745205] upgrade[6905]: [508 /779 ] (51%) installing usbredir-0.7.1-3.el7... [ 112.780151] upgrade[6905]: [509 /779 ] (51%) installing uptrack-offline-1.2.62.offline-0.el7... [ 113.001966] upgrade[6905]: [510 /779 ] (51%) installing dconf-0.28.0-4.el7... [ 113.129938] upgrade[6905]: [511 /779 ] (51%) installing iscsi-initiator-utils-iscsiuio-6.2.0.874-11.0.1.el7... [ 113.178038] upgrade[6905]: [512 /779 ] (51%) installing iscsi-initiator-utils-6.2.0.874-11.0.1.el7... [ 113.362242] upgrade[6905]: [513 /779 ] (52%) installing unbound-libs-1.6.6-1.el7... |
At the end of the process the server will reboot. And if everything is fine you should have an Linux 7.7 now
1
2
3
|
[root@oda-server~] # cat /etc/oracle-release Oracle Linux Server release 7.7 [root@oda-server~] # |
After the operating system upgrade is completed successfully, run the post upgrade checks:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
[root@oda-server~] # /opt/oracle/dcs/bin/odacli update-server-postcheck -v 19.6.0.0.0 Upgrade post-check report ------------------------- Node Name --------------- server-oda Comp Pre-Check Status Comments ----- ------------------------------ -------- -------------------------------------- OS OS upgrade check SUCCESS OS has been upgraded to OL7 GI GI upgrade check INFO GI home needs to update to 19.6.0.0.200114 GI GI status check SUCCESS Clusterware is running on the node OS ODABR snapshot WARNING ODABR snapshot found. Run 'odabr delsnap' to delete. RPM Extra RPM check SUCCESS No extra RPMs found when OS was at OL6 [root@oda-server~] # |
As specified let’s remove the snapshots we took with odabr
1
2
3
4
5
6
7
8
9
10
11
|
[root@oda-server~] # /opt/odabr/odabr delsnap INFO: 2020-05-29 14:22:31: Please check the logfile '/opt/odabr/out/log/odabr_60717.log' for more details INFO: 2020-05-29 14:22:31: Removing LVM snapshots INFO: 2020-05-29 14:22:31: ...removing LVM snapshot for 'opt' SUCCESS: 2020-05-29 14:22:32: ...snapshot for 'opt' removed successfully INFO: 2020-05-29 14:22:32: ...removing LVM snapshot for 'u01' SUCCESS: 2020-05-29 14:22:32: ...snapshot for 'u01' removed successfully INFO: 2020-05-29 14:22:32: ...removing LVM snapshot for 'root' SUCCESS: 2020-05-29 14:22:32: ...snapshot for 'root' removed successfully SUCCESS: 2020-05-29 14:22:32: Remove LVM snapshots done successfully |
Running the posthecks again
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
[root@oda-server~] # /opt/oracle/dcs/bin/odacli update-server-postcheck -v 19.6.0.0.0 Upgrade post-check report ------------------------- Node Name --------------- server-oda Comp Pre-Check Status Comments ----- ------------------------------ -------- -------------------------------------- OS OS upgrade check SUCCESS OS has been upgraded to OL7 GI GI upgrade check INFO GI home needs to update to 19.6.0.0.200114 GI GI status check SUCCESS Clusterware is running on the node OS ODABR snapshot SUCCESS No ODABR snapshots found RPM Extra RPM check SUCCESS No extra RPMs found when OS was at OL6 [root@oda-server~] # |
After the OS upgrade now let’s upgrade the remaining components
We start by updating the DSC agent
1
2
3
4
5
6
7
8
9
10
11
12
|
[root@oda-server~] # /opt/oracle/dcs/bin/odacli update-dcsadmin -v 19.6.0.0.0 { "jobId" : "5bdb8d99-d3ae-421c-a05d-60b9ece94021" , "status" : "Created" , "message" : null, "reports" : [ ], "createTimestamp" : "May 29, 2020 14:26:32 PM CEST" , "resourceList" : [ ], "description" : "DcsAdmin patching" , "updatedTime" : "May 29, 2020 14:26:32 PM CEST" } [root@oda-server~] # |
We validate that the Job returns success
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
[root@oda-server~] # /opt/oracle/dcs/bin/odacli describe-job -i "5bdb8d99-d3ae-421c-a05d-60b9ece94021" Job details ---------------------------------------------------------------- ID: 5bdb8d99-d3ae-421c-a05d-60b9ece94021 Description: DcsAdmin patching Status: Success Created: May 29, 2020 2:26:32 PM CEST Message: Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- Patch location validation May 29, 2020 2:26:32 PM CEST May 29, 2020 2:26:32 PM CEST Success dcsadmin upgrade May 29, 2020 2:26:32 PM CEST May 29, 2020 2:26:32 PM CEST Success Update System version May 29, 2020 2:26:32 PM CEST May 29, 2020 2:26:32 PM CEST Success [root@oda-server~] # |
We update the DCS components
1
2
3
4
5
6
7
8
9
10
11
|
[root@oda-server~] # /opt/oracle/dcs/bin/odacli update-dcscomponents -v 19.6.0.0.0 { "jobId" : "4af0626e-d260-4f93-8948-2f241f7b2c48" , "status" : "Success" , "message" : null, "reports" : null, "createTimestamp" : "May 29, 2020 14:27:52 PM CEST" , "description" : "Job completed and is not part of Agent job list" , "updatedTime" : "May 29, 2020 14:27:52 PM CEST" } [root@oda-server~] # |
And then we update the server
1
2
3
4
5
6
7
8
9
10
11
12
|
[root@oda-server~] # /opt/oracle/dcs/bin/odacli update-server -v 19.6.0.0.0 { "jobId" : "91f7b206-b097-4ba2-bae7-42a6b3bb0ba6" , "status" : "Created" , "message" : "Success of server update will trigger reboot of the node after 4-5 minutes. Please wait until the node reboots." , "reports" : [ ], "createTimestamp" : "May 29, 2020 14:31:25 PM CEST" , "resourceList" : [ ], "description" : "Server Patching" , "updatedTime" : "May 29, 2020 14:31:25 PM CEST" } [root@oda-server~] # |
This will lasts about 45 minutes. Validate that all is fine
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
|
[root@oda-server~] # /opt/oracle/dcs/bin/odacli describe-job -i "91f7b206-b097-4ba2-bae7-42a6b3bb0ba6" Job details ---------------------------------------------------------------- ID: 91f7b206-b097-4ba2-bae7-42a6b3bb0ba6 Description: Server Patching Status: Success Created: May 29, 2020 2:31:25 PM CEST Message: Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- Patch location validation May 29, 2020 2:31:33 PM CEST May 29, 2020 2:31:33 PM CEST Success dcs-controller upgrade May 29, 2020 2:31:33 PM CEST May 29, 2020 2:31:33 PM CEST Success Creating repositories using yum May 29, 2020 2:31:33 PM CEST May 29, 2020 2:31:34 PM CEST Success Applying HMP Patches May 29, 2020 2:31:34 PM CEST May 29, 2020 2:31:34 PM CEST Success Patch location validation May 29, 2020 2:31:34 PM CEST May 29, 2020 2:31:34 PM CEST Success oda-hw-mgmt upgrade May 29, 2020 2:31:35 PM CEST May 29, 2020 2:31:35 PM CEST Success OSS Patching May 29, 2020 2:31:35 PM CEST May 29, 2020 2:31:35 PM CEST Success Applying Firmware Disk Patches May 29, 2020 2:31:35 PM CEST May 29, 2020 2:31:40 PM CEST Success Applying Firmware Expander Patches May 29, 2020 2:31:40 PM CEST May 29, 2020 2:31:44 PM CEST Success Applying Firmware Controller Patches May 29, 2020 2:31:44 PM CEST May 29, 2020 2:31:48 PM CEST Success Checking Ilom patch Version May 29, 2020 2:31:50 PM CEST May 29, 2020 2:31:52 PM CEST Success Patch location validation May 29, 2020 2:31:52 PM CEST May 29, 2020 2:31:53 PM CEST Success Save password in Wallet May 29, 2020 2:31:54 PM CEST May 29, 2020 2:31:54 PM CEST Success Apply Ilom patch May 29, 2020 2:31:54 PM CEST May 29, 2020 2:39:55 PM CEST Success Copying Flash Bios to Temp location May 29, 2020 2:39:55 PM CEST May 29, 2020 2:39:55 PM CEST Success Starting the clusterware May 29, 2020 2:39:55 PM CEST May 29, 2020 2:41:52 PM CEST Success Creating GI home directories May 29, 2020 2:41:53 PM CEST May 29, 2020 2:41:53 PM CEST Success Cloning Gi home May 29, 2020 2:41:53 PM CEST May 29, 2020 2:44:16 PM CEST Success Configuring GI May 29, 2020 2:44:16 PM CEST May 29, 2020 2:46:25 PM CEST Success Running GI upgrade root scripts May 29, 2020 2:46:25 PM CEST May 29, 2020 3:03:15 PM CEST Success Resetting DG compatibility May 29, 2020 3:03:15 PM CEST May 29, 2020 3:03:33 PM CEST Success Running GI config assistants May 29, 2020 3:03:33 PM CEST May 29, 2020 3:04:18 PM CEST Success restart oakd May 29, 2020 3:04:28 PM CEST May 29, 2020 3:04:39 PM CEST Success Updating GiHome version May 29, 2020 3:04:39 PM CEST May 29, 2020 3:04:45 PM CEST Success Update System version May 29, 2020 3:04:54 PM CEST May 29, 2020 3:04:54 PM CEST Success preRebootNode Actions May 29, 2020 3:04:54 PM CEST May 29, 2020 3:05:40 PM CEST Success Reboot Ilom May 29, 2020 3:05:40 PM CEST May 29, 2020 3:05:40 PM CEST Success [root@oda-server~] # |
And we can verify that the components were upgraded
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
[root@oda-server~] # /opt/oracle/dcs/bin/odacli describe-component System Version --------------- 19.6.0.0.0 Component Installed Version Available Version ---------------------------------------- -------------------- -------------------- OAK 19.6.0.0.0 up-to- date GI 19.6.0.0.200114 up-to- date DB 11.2.0.4.190115 11.2.0.4.200114 DCSAGENT 19.6.0.0.0 up-to- date ILOM 4.0.4.52.r133103 up-to- date BIOS 41060700 up-to- date OS 7.7 up-to- date FIRMWARECONTROLLER QDV1RF30 up-to- date FIRMWAREDISK 0121 up-to- date HMP 2.4.5.0.1 up-to- date [root@oda-server~] # |
After the server we have to update the storage
1
2
3
4
5
6
7
8
9
10
11
12
|
[root@oda-server~] # /opt/oracle/dcs/bin/odacli update-storage -v 19.6.0.0.0 { "jobId" : "c4b365ff-bd8c-4bca-b53d-d9d9bda90548" , "status" : "Created" , "message" : "Success of Storage Update may trigger reboot of node after 4-5 minutes. Please wait till node restart" , "reports" : [ ], "createTimestamp" : "May 29, 2020 15:16:17 PM CEST" , "resourceList" : [ ], "description" : "Storage Firmware Patching" , "updatedTime" : "May 29, 2020 15:16:17 PM CEST" } [root@oda-server~] # |
We can verify that the job was successful
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
[root@oda-server~] # /opt/oracle/dcs/bin/odacli describe-job -i "c4b365ff-bd8c-4bca-b53d-d9d9bda90548" Job details ---------------------------------------------------------------- ID: c4b365ff-bd8c-4bca-b53d-d9d9bda90548 Description: Storage Firmware Patching Status: Success Created: May 29, 2020 3:16:17 PM CEST Message: Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- Applying Firmware Disk Patches May 29, 2020 3:16:18 PM CEST May 29, 2020 3:16:21 PM CEST Success Applying Firmware Controller Patches May 29, 2020 3:16:21 PM CEST May 29, 2020 3:16:26 PM CEST Success preRebootNode Actions May 29, 2020 3:16:26 PM CEST May 29, 2020 3:16:26 PM CEST Success Reboot Ilom May 29, 2020 3:16:26 PM CEST May 29, 2020 3:16:26 PM CEST Success [root@oda-server~] # |
1
|
<br> |
I did not patch the existing database home because I just want to keep the actual version for my existing databases.
To be able to create new 19c databases we have to update the repository with corresponding rdbms clone.
After unzipping the archive
1
2
3
|
oracle@server-oda: /u01/oda_patch_mdi/19 .6/ [ORCL] unzip p30403662_196000_Linux-x86-64.zip Archive: p30403662_196000_Linux-x86-64.zip extracting: odacli-dcs-19.6.0.0.0-200326-DB-19.6.0.0.zip |
I just have to update the repository with the db clone
1
2
3
4
5
6
7
8
9
10
11
12
|
[[email protected]] # /opt/oracle/dcs/bin/odacli update-repository -f /u01/oda_patch_mdi/19.6/odacli-dcs-19.6.0.0.0-200326-DB-19.6.0.0.zip { "jobId" : "e8e58467-460f-4b07-86db-d4b71f1e5884" , "status" : "Created" , "message" : "/u01/oda_patch_mdi/19.6/odacli-dcs-19.6.0.0.0-200326-DB-19.6.0.0.zip" , "reports" : [ ], "createTimestamp" : "May 29, 2020 15:23:37 PM CEST" , "resourceList" : [ ], "description" : "Repository Update" , "updatedTime" : "May 29, 2020 15:23:37 PM CEST" } |
Then I check that the job was fine
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
[[email protected]] # /opt/oracle/dcs/bin/odacli describe-job -i "e8e58467-460f-4b07-86db-d4b71f1e5884" Job details ---------------------------------------------------------------- ID: e8e58467-460f-4b07-86db-d4b71f1e5884 Description: Repository Update Status: Success Created: May 29, 2020 3:23:37 PM CEST Message: /u01/oda_patch_mdi/19 .6 /odacli-dcs-19 .6.0.0.0-200326-DB-19.6.0.0.zip Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- |
Encountered ACFS Issue
During the patch I did not have any issue. All was fine. But at the end of the patch my existing databases did not come up because of an acfs issue.
Indeed, the ASM instance was up but the ASM proxy instance APX was instable
1
2
3
4
5
6
7
8
|
[grid@oda-server~]$ srvctl start asm -proxy -node oda-server PRCR-1013 : Failed to start resource ora.proxy_advm PRCR-1064 : Failed to start resource ora.proxy_advm on node oda-server CRS-5017: The resource action "ora.proxy_advm start" encountered the following error: ORA-01092: ORACLE instance terminated. Disconnection forced Process ID: 0 Session ID: 0 Serial number: 0 . For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/oda-server/crs/trace/crsd_oraagent_grid.trc" . |
Looking on the trace file
1
2
3
4
5
6
7
8
9
10
11
12
|
[root@oda-servertrace] # grep -i ORA- /u01/app/grid/diag/crs/oda-server/crs/trace/crsd_oraagent_grid.trc 2020-05-29 16:51:18.216 :CLSDYNAM:916408064: [ora.proxy_advm]{1:38257:2942} [start] ORA-03113: end-of- file on communication channel 2020-05-29 16:51:18.216 :CLSDYNAM:916408064: [ora.proxy_advm]{1:38257:2942} [start] InstAgent::startInstance 250 ORA-3113retryCount:0 m_instanceType:2 m_lastOCIError:3113 2020-05-29 16:51:39.641 :CLSDYNAM:916408064: [ora.proxy_advm]{1:38257:2942} [start] ORA-03113: end-of- file on communication channel 2020-05-29 16:51:39.641 :CLSDYNAM:916408064: [ora.proxy_advm]{1:38257:2942} [start] InstAgent::startInstance 250 ORA-3113retryCount:1 m_instanceType:2 m_lastOCIError:3113 2020-05-29 16:52:01.078 :CLSDYNAM:916408064: [ora.proxy_advm]{1:38257:2942} [start] ORA-03113: end-of- file on communication channel 2020-05-29 16:52:01.078 :CLSDYNAM:916408064: [ora.proxy_advm]{1:38257:2942} [start] InstAgent::startInstance 250 ORA-3113retryCount:2 m_instanceType:2 m_lastOCIError:3113 2020-05-29 16:52:01.098 :CLSDYNAM:916408064: [ora.proxy_advm]{1:38257:2942} [start] ORA-01092: ORACLE instance terminated. Disconnection forced 2020-05-29 16:52:01.099 :CLSDYNAM:916408064: [ora.proxy_advm]{1:38257:2942} [start] InstAgent::startInstance 380 throw excp what:ORA-01092: ORACLE instance terminated. Disconnection forced 2020-05-29 16:52:01.099 :CLSDYNAM:916408064: [ora.proxy_advm]{1:38257:2942} [start] ORA-01092: ORACLE instance terminated. Disconnection forced ORA-01092: ORACLE instance terminated. Disconnection forced [root@oda-servertrace] # |
For some reason the acfs module was not loaded. My databases was Oracle 11.2.0.4 and were using acfs.
1
2
3
|
[root@oda-server~] # /sbin/lsmod | grep oracle oracleoks 724992 0 oracleafd 229376 0 |
So I first stop my crs
1
|
[root@oda-server~] # crsctl stop crs |
And then I reinstall the acfs module with following acfsroot command
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
[root@oda-server~] # which acfsroot /u01/app/19 .0.0.0 /grid/bin/acfsroot [root@oda-server~] # /u01/app/19.0.0.0/grid/bin/acfsroot install ACFS-9300: ADVM /ACFS distribution files found. ACFS-9314: Removing previous ADVM /ACFS installation. ACFS-9315: Previous ADVM /ACFS components successfully removed. ACFS-9294: updating file /etc/sysconfig/oracledrivers .conf ACFS-9307: Installing requested ADVM /ACFS software. ACFS-9294: updating file /etc/sysconfig/oracledrivers .conf ACFS-9308: Loading installed ADVM /ACFS drivers. ACFS-9321: Creating udev for ADVM /ACFS . ACFS-9323: Creating module dependencies - this may take some time . ACFS-9154: Loading 'oracleoks.ko' driver. ACFS-9154: Loading 'oracleadvm.ko' driver. ACFS-9154: Loading 'oracleacfs.ko' driver. ACFS-9327: Verifying ADVM /ACFS devices. ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec' . ACFS-9156: Detecting control device '/dev/ofsctl' . ACFS-9309: ADVM /ACFS installation correctness verified. [root@oda-server~] # |
Running again the lsmod command, I see that things are better
1
2
3
4
5
6
|
[root@oda-server~] # /sbin/lsmod | grep oracle oracleacfs 5828608 0 oracleadvm 1204224 0 oracleoks 724992 2 oracleadvm,oracleacfs oracleafd 229376 0 [root@oda-server~] # |
After I reboot my server and all databases and the ASM Proxy Instance become up and running. I just wanted to mention this issue I had and hope this will help.
Conclusion
Some recommendations
Remove all rpm packages that were manually installed
Verify that you have enough space on / , /u01 and /opt filesystems. /opt and /u01 can be increased online
Launch the OS upgrade from the ILOM console
I hope this blog will help.