Introduction

ODA X8-2, in the S, M or HA flavour, is the new database appliance from Oracle. But as a brand new product, you can experience troubles due to the lack of maturity. This time, it concerns database creation that can be extremely slow on these servers if you are using ODA 19.5.

Edit (April 2020): actually this bug is related to the Linux kernel. There will not be a patch to solve the problem. 19.6 should have a new kernel without this bug.

Deployment, appliance creation and core configuration

X8-2 reimaging is fine, and appliance creation is done without any problem on X8-2. Don’t hesitate to create a DBTEST database during appliance creation, it’s a good practise to check if everything is OK after deployment. Once appliance creation is done you should find these pmon processes running on the system:

[root@oda-cr-test ~]# ps -ef | grep pmon
oracle 17600 1 0 11:49 ? 00:00:00 asm_pmon_+ASM1
oracle 20604 1 0 11:50 ? 00:00:00 apx_pmon_+APX1
oracle 21651 1 0 11:50 ? 00:00:00 ora_pmon_DBTEST
root 81984 37658 0 13:44 pts/0 00:00:00 grep --color=auto pmon

If you deployed your ODA for Enterprise Edition, you should now apply the core configuration, understand disabling the unlicensed cores. It’s better to do that immediatly after deployment to avoid using cores you didn’t pay for. One EE license is 2 cores, 2 licenses is 4 cores, etc:

odacli update-cpucore -c 2

Core configuration is done within a few seconds, you could check with this command:

odacli describe-cpucore
Node Cores Modified Job Status
----- ------ ------------------------------ ---------------
0 2 January 27, 2020 9:56:37 AM CET Configured

Starting from now, you can create your own databases.

Database creation: extremely slow and eventually failing

Without additional steps you can create your very first DB. I’m using basic shape for all my databases, as I fine tune everything later using a SQL script (SGA and PGA targets, undo_retention, archive_lag_target, redolog configuration, options, etc):

[root@oda-cr-test ~]# odacli list-dbhomes
ID Name DB Version Home Location Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
ae442886-6bcc-497c-8e16-2e8b4e55157e OraDB19000_home1 19.5.0.0.191015 /u01/app/oracle/product/19.0.0.0/dbhome_1 Configured
odacli create-database -m MAnager_2020_dbi -cs AL32UTF8 -no-c -u DBSUS -dh 'ae442886-6bcc-497c-8e16-2e8b4e55157e' -n DBSUS -s odb1s -l AMERICAN -dt AMERICA -no-co -r asm

ASM is a better solution than acfs for me: no need to have “real” filesystems and optimal storage usage.

Creating the database lasts normally less than 10 minutes, but not this time:


odacli describe-job -i 94726b11-ea0c-46b0-ab55-6d709ef747d3
Job details
----------------------------------------------------------------
ID: 94726b11-ea0c-46b0-ab55-6d709ef747d3
Description: Database service creation with db name: DBSUS
Status: Failure
Created: February 13, 2020 11:24:53 AM CET
Message: DCS-10001:Internal error encountered: Fail to create User tablespace .
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Database Service creation February 13, 2020 11:24:53 AM CET February 13, 2020 6:23:16 PM CET Failure
Database Service creation February 13, 2020 11:24:53 AM CET February 13, 2020 6:23:16 PM CET Failure
Setting up ssh equivalance February 13, 2020 11:26:26 AM CET February 13, 2020 11:26:36 AM CET Success
Database Service creation February 13, 2020 11:26:36 AM CET February 13, 2020 6:23:11 PM CET Success
Database Creation February 13, 2020 11:26:36 AM CET February 13, 2020 6:21:37 PM CET Success
Change permission for xdb wallet files February 13, 2020 6:21:37 PM CET February 13, 2020 6:21:38 PM CET Success
Place SnapshotCtrlFile in sharedLoc February 13, 2020 6:21:38 PM CET February 13, 2020 6:22:06 PM CET Success
SqlPatch upgrade February 13, 2020 6:23:02 PM CET February 13, 2020 6:23:05 PM CET Success
Running dbms_stats init_package February 13, 2020 6:23:05 PM CET February 13, 2020 6:23:07 PM CET Success
updating the Database version February 13, 2020 6:23:07 PM CET February 13, 2020 6:23:11 PM CET Success
create Users tablespace February 13, 2020 6:23:11 PM CET February 13, 2020 6:23:16 PM CET Failure
Creating Users tablespace February 13, 2020 6:23:11 PM CET February 13, 2020 6:23:16 PM CET Failure

I took 7 hours and it finished with a failure!

Failure can be slightly different, for example :
DCS-10001:Internal error encountered: configure snapshot control file for databaseDBSUS.

Troubleshooting

Creation of the DBTEST database was fine at appliance creation. So what changed between deployment and database creation? The cpu-core configuration. Something is probably wrong with this core configuration.

dbca log is not very interesting regarding our problem:

cat /u01/app/oracle/cfgtoollogs/dbca/DBSUS/DBSUS.log
[ 2020-02-13 11:37:46.492 CET ] Prepare for db operation
DBCA_PROGRESS : 10%
[ 2020-02-13 11:38:22.441 CET ] Registering database with Oracle Restart
DBCA_PROGRESS : 14%
[ 2020-02-13 11:38:51.365 CET ] Copying database files
DBCA_PROGRESS : 43%
[ 2020-02-13 12:09:44.044 CET ] Creating and starting Oracle instance
DBCA_PROGRESS : 45%
DBCA_PROGRESS : 49%
DBCA_PROGRESS : 53%
DBCA_PROGRESS : 56%
DBCA_PROGRESS : 62%
[ 2020-02-13 15:44:47.163 CET ] Completing Database Creation
DBCA_PROGRESS : 68%
[ 2020-02-13 17:48:28.024 CET ] [WARNING] ORA-13516: AWR Operation failed: AWR Schema not initialized
ORA-06512: at "SYS.DBMS_SWRF_INTERNAL", line 356
ORA-06512: at "SYS.DBMS_SWRF_INTERNAL", line 389
ORA-06512: at line 1
DBCA_PROGRESS : 71%
[ 2020-02-13 18:20:49.119 CET ] Executing Post Configuration Actions
DBCA_PROGRESS : 100%
[ 2020-02-13 18:20:49.123 CET ] Database creation complete. For details check the logfiles at:
/u01/app/oracle/cfgtoollogs/dbca/DBSUS.
Database Information:
Global Database Name:DBSUS.salt.ch
System Identifier(SID):DBSUS

Let’s check the ASM alert.log:

tail /u01/app/oracle/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log
2020-02-13T14:35:24.880987+01:00
LGWR (ospid: 17755) waits for event 'kfk: async disk IO' for 44 secs.
2020-02-13T14:35:24.881028+01:00
LGWR (ospid: 17755) is hung in an acceptable location (inwait 0x1.ffff).
2020-02-13T14:35:30.039917+01:00
LGWR (ospid: 17755) waits for event 'kfk: async disk IO' for 59 secs.
2020-02-13T14:35:40.280979+01:00
LGWR (ospid: 17755) waits for event 'kfk: async disk IO' for 75 secs.
2020-02-13T14:37:13.180477+01:00
LGWR (ospid: 17755) waits for event 'kfk: async disk IO' for 39 secs.
2020-02-13T14:37:13.180536+01:00
LGWR (ospid: 17755) is hung in an acceptable location (inwait 0x1.ffff).

There is a problem with the disks.

What the system tells us?

tail /var/log/messages
...
Feb 13 14:34:49 oda-cr-test kernel: nvme nvme1: I/O 615 QID 1 timeout, completion polled
Feb 13 14:35:09 oda-cr-test kernel: nvme nvme0: I/O 348 QID 1 timeout, completion polled
Feb 13 14:35:12 oda-cr-test kernel: nvme nvme1: I/O 623 QID 1 timeout, completion polled
Feb 13 14:35:12 oda-cr-test kernel: nvme nvme1: I/O 624 QID 1 timeout, completion polled
Feb 13 14:35:16 oda-cr-test kernel: nvme nvme0: I/O 349 QID 1 timeout, completion polled
Feb 13 14:35:40 oda-cr-test kernel: nvme nvme0: I/O 348 QID 1 timeout, completion polled
Feb 13 14:35:46 oda-cr-test kernel: nvme nvme0: I/O 349 QID 1 timeout, completion polled
Feb 13 14:35:46 oda-cr-test kernel: nvme nvme0: I/O 351 QID 1 timeout, completion polled
Feb 13 14:35:46 oda-cr-test kernel: nvme nvme0: I/O 352 QID 1 timeout, completion polled
Feb 13 14:35:46 oda-cr-test kernel: nvme nvme0: I/O 353 QID 1 timeout, completion polled
Feb 13 14:35:46 oda-cr-test kernel: nvme nvme0: I/O 354 QID 1 timeout, completion polled
...

Let’s try to create a database with more cores:

odacli update-cpucore -c 16
reboot
odacli create-database -m MAnager_2020_dbi -cs AL32UTF8 -no-c -u DBSUP -dh '2d147842-4f42-468a-93c9-112ce9c23ee7' -n DBSUP -s odb1s -l AMERICAN -dt AMERICA -no-co -r asm
odacli describe-job -i 204fcd53-a9f1-416e-953b-c50448207fc1

Job details
----------------------------------------------------------------
ID: 204fcd53-a9f1-416e-953b-c50448207fc1
Description: Database service creation with db name: DBSUP
Status: Success
Created: February 7, 2020 3:41:54 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Setting up ssh equivalance February 7, 2020 3:41:55 PM CET February 7, 2020 3:41:55 PM CET Success
Database Service creation February 7, 2020 3:41:56 PM CET February 7, 2020 3:50:07 PM CET Success
Database Creation February 7, 2020 3:41:56 PM CET February 7, 2020 3:48:16 PM CET Success
Change permission for xdb wallet files February 7, 2020 3:48:16 PM CET February 7, 2020 3:48:16 PM CET Success
Place SnapshotCtrlFile in sharedLoc February 7, 2020 3:48:16 PM CET February 7, 2020 3:48:19 PM CET Success
SqlPatch upgrade February 7, 2020 3:49:52 PM CET February 7, 2020 3:50:01 PM CET Success
Running dbms_stats init_package February 7, 2020 3:50:01 PM CET February 7, 2020 3:50:05 PM CET Success
updating the Database version February 7, 2020 3:50:05 PM CET February 7, 2020 3:50:07 PM CET Success
create Users tablespace February 7, 2020 3:50:07 PM CET February 7, 2020 3:50:12 PM CET Success
Clear all listeners from Databse {8a6b0534-26be-4a9a-90fd-f2167f57fded} February 7, 2020 3:50:12 PM CET February 7, 2020 3:50:14 PM CET Success

Success in 9 minutes.

odacli update-cpucore -c 8 --force
reboot
odacli create-database -m MAnager_2020_dbi -cs AL32UTF8 -no-c -u DBSUQ -dh '2d147842-4f42-468a-93c9-112ce9c23ee7' -n DBSUQ -s odb1s -l AMERICAN -dt AMERICA -no-co -r asm
odacli describe-job -i 210e1bd7-fe87-4d60-adce-84b121448c2b

Job details
----------------------------------------------------------------
ID: 210e1bd7-fe87-4d60-adce-84b121448c2b
Description: Database service creation with db name: DBSUQ
Status: Success
Created: February 7, 2020 4:05:48 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Setting up ssh equivalance February 7, 2020 4:05:49 PM CET February 7, 2020 4:05:49 PM CET Success
Database Service creation February 7, 2020 4:05:49 PM CET February 7, 2020 5:53:09 PM CET Success
Database Creation February 7, 2020 4:05:49 PM CET February 7, 2020 5:46:50 PM CET Success
Change permission for xdb wallet files February 7, 2020 5:46:50 PM CET February 7, 2020 5:46:50 PM CET Success
Place SnapshotCtrlFile in sharedLoc February 7, 2020 5:46:50 PM CET February 7, 2020 5:48:55 PM CET Success
SqlPatch upgrade February 7, 2020 5:50:28 PM CET February 7, 2020 5:53:02 PM CET Success
Running dbms_stats init_package February 7, 2020 5:53:02 PM CET February 7, 2020 5:53:06 PM CET Success
updating the Database version February 7, 2020 5:53:06 PM CET February 7, 2020 5:53:09 PM CET Success
create Users tablespace February 7, 2020 5:53:09 PM CET February 7, 2020 5:53:45 PM CET Success
Clear all listeners from Databse {f0900b63-baf8-4896-8572-a4120770a362} February 7, 2020 5:53:45 PM CET February 7, 2020 5:53:47 PM CET Success

Success in 1h50.


odacli update-cpucore -c 4 --force
reboot
odacli create-database -m MAnager_2020_dbi -cs AL32UTF8 -no-c -u DBSUR -dh '2d147842-4f42-468a-93c9-112ce9c23ee7' -n DBSUR -s odb1s -l AMERICAN -dt AMERICA -no-co -r asm
odacli describe-job -i 1931b118-3407-413b-babc-ff9a832fab59

Job details
----------------------------------------------------------------
ID: 1931b118-3407-413b-babc-ff9a832fab59
Description: Database service creation with db name: DBSUR
Status: Success
Created: February 7, 2020 6:25:37 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Setting up ssh equivalance February 7, 2020 6:25:38 PM CET February 7, 2020 6:25:38 PM CET Success
Database Service creation February 7, 2020 6:25:38 PM CET February 7, 2020 10:09:17 PM CET Success
Database Creation February 7, 2020 6:25:38 PM CET February 7, 2020 9:30:40 PM CET Success
Change permission for xdb wallet files February 7, 2020 9:30:40 PM CET February 7, 2020 9:30:40 PM CET Success
Place SnapshotCtrlFile in sharedLoc February 7, 2020 9:30:40 PM CET February 7, 2020 9:35:21 PM CET Success
SqlPatch upgrade February 7, 2020 9:51:49 PM CET February 7, 2020 10:08:17 PM CET Success
Running dbms_stats init_package February 7, 2020 10:08:17 PM CET February 7, 2020 10:09:14 PM CET Success
updating the Database version February 7, 2020 10:09:14 PM CET February 7, 2020 10:09:17 PM CET Success
create Users tablespace February 7, 2020 10:09:17 PM CET February 7, 2020 10:39:01 PM CET Success
Clear all listeners from Databse {8cb507cf-3c84-4ad9-8302-844005965f6b} February 7, 2020 10:39:01 PM CET February 7, 2020 10:41:49 PM CET Success

Success in 4h15.

That’s it. Decreasing the cores dramatically decreases the I/O performance and makes our ODA unusable.

And this problem doesn’t limit to database creation: don’t expect to run correctly a database with this bug. A small datafile creation will take long minutes.

Are you concerned?

This problem does not concern everybody. If you use Standard Edition 2 databases, you don’t need to decrease the cores on your ODA. So you won’t experience this problem. If you have enough Enterprise Edition licenses (at least to enable half of the total cores), you also won’t have this problem. This problem only impacts those who have limited number of licenses.

All my tests were done with ODA 19.5, so it’s probably limited to this specific version. ODA X8.2 with 18.7 shouldn’t have this bug.

Is there a workaround?

For now, there is no workaround. But it seems that an updated Linux kernel could solve the problem. Current kernel provided with 19.5 is 4.14.35-1902.5.2.el7uek.x86_64. Oracle will probably provide something soon.

Edit (April 2020): No new kernel or fix will be provided by Oracle for this 19.5 version. The only way to solve the problem is going for 19.6 (most likely to be released in May).

Conclusion

One of the main advantages of ODA is the core configuration to fit the license. But for now it doesn’t work out of the box with ODA X8-2 and 19.5. If you can keep 11g/12c/18c, waiting for release 19.6 is probably better.