Oracle Database Appliance: which storage capacity to choose?

Introduction

If you’re considering ODA for your next platform, you surely already appreciate the simplicity of the offer. 3 models with few options, this is definitely easy to choose from.

One of the other benefit is also the hardware support of 5 years, and combined with software updates generally available for up to 7 years old ODAs, you can keep your ODA running even longer for non-critical databases and/or if you have a strong Disaster Recovery solution (including Data Guard or Dbvisit standby). Some of my customers are still using X4-2 models and are confident in their ODAs because it’s been quite reliable across the years.

Models and storage limits

One of the main drawback of the ODA: it doesn’t have unlimited storage. Disks are local NVMe SSDs (or in a dedicated enclosure), and it’s not possible (technically possible but not recommended) to add storage through a NAS connection.

3 ODA models are available, X8-2S and X8-2M are one-node ODAs, and X8-2HA being a two-nodes ODA with DAS storage including SSD and/or HDD (High Performance or High Capacity version).

Please refer to my previous blog post for more information about the current generation.

Storage on ODA is always dedicated to database related files: datafiles, redologs, controlfiles, archivelogs, flashback logs, backups (if you do it locally on ODA), etc. Linux system, Oracle products (Grid Infrastructure and Oracle database engines), home folders and so on reside on internal M2 SSD disks large enough for a normal use.

X8-2S/X8-2M storage limit

ODA X8-2S is the entry level ODA. It only has one CPU, but with 16 powerful cores and 192GB of memory it’s all but a low end server. 10 empty storage slots are available in the front pane but don’t expect to extend the storage. This ODA is delivered with 2 disks and doesn’t support adding more disks. That’s it. With the two 6.4TB disks, you’ll have a RAW capacity of 12.8TB.

ODA X8-2M is much more capable than his little brother. Physically identical to X8-2S, it has two CPUs and twice the amount of RAM. This 32-cores server fitted with 384GB of RAM is a serious player. It’s still delivered with two 6.4TB disks but unlike the S version, all the 10 empty storage slots can be populated to reach a stunning 76.8TB of RAW storage. This is still not unlimited, but the limit is actually quite high. Disks can be added by pair, so you can have 2-4-6-8-10-12 disks for various configurations and for a maximum of 76.8TB RAW capacity. Only disks dedicated for ODA are suitable, and don’t expect to put bigger disks as it only supports the same 6.4TB disks than those embedded with the base server.

RAW capacity means without redundancy, and you will loose half of the capacity with ASM redundancy. It’s not possible to run an ODA without redundancy, if you think about that. ASM redundancy is the only way to secure data, as no RAID controller is inside the server. You already know that disk capacity and real capacity always differs, so Oracle included several years ago in the documentation the usable capacity depending on your configuration. The usable capacity includes reserved space for a single disk failure (15% starting from 4 disks).

On base ODAs (X8-2S and X8-2M with 2 disks only). The usable storage capacity is actually 5.8TB and no space is reserved for disk failure. If a disk fails, there is no way to rebuild redundancy as only one disk survives.

Usable storage is not database storage, don’t miss that point. You’ll need to split this usable storage between DATA area and RECO area (actually ASM diskgroups). Most often, RECO is sized between 10% and 30% of usable storage.

Here is a table with various configurations. Note that I didn’t include ASM high redundancy configurations here, I’ll explain that later.

X8-2HA storage limit

Storage is more complex on X8-2HA. If you’re looking for complete information about its storage, review the ODA documentation for all the possibilities.

Briefly, X8-HA is available in two flavors: High Performance, the one I highly recommend, or High Capacity, which is nice if you have really big databases you want to store on only one ODA. But this High Capacity version will make use of spinning disks to achieve such amount of TB. Definitely not the best solution for the performance. The 2 nodes of this ODA are empty, no disk in the front panel, just empty space. All data disks are in a separate enclosure connected on both nodes with SAS cables. Depending on your configuration, you’ll have 6 to 24 SSD (HP) or a mix of 6 SSD and 18 HDD (HC). When your first enclosure is filled with disks, you can also add another storage enclosure of the same kind to eventually double the total capacity. Usable storage starts from 17.8TB to 142.5TB for HP, and from 114.8TB to 230.6TB for HC.

Best practice for storage usage

First you should consider that ODA storage is high performance storage for high database throughput. Thus, storing backups on ODA is a nonsense. Backups are files written once and mainly dedicated to get erased without being used. Don’t loose precious TB for that. Moreover, if backups are done in the FRA, they are actually located on the same disks as DATA. It’s why most of the configuration will be done with 10% to 20% of RECO, not more. Because we definitely won’t put backups on the same disks as DATA. 10% for RECO is a minimum, I wouldn’t recommend setting less than that, Fast Recovery Area being always a problem if too small.

During deployment you’ll have to choose between NORMAL or HIGH redundancy. NORMAL is quite similar to RAID1, but at the block level and without requiring disk parity (you need 2 or more disks). HIGH is available starting from 3 disks and makes each block existing 3 times on 3 different disks. HIGH seems to be better, but you loose even more precious space, and it doesn’t protect you from other failures like disaster in your datacenter or user errors. Most of the failure protection systems embedded in the servers are actually doubling the components: power supplies, network interfaces, system disks, and so on. So increasing the security of block redundancy without increasing the security of other components is not necessary in my opinion. Real solution for increased failure protection is Data Guard or Dbvisit: 2 ODAs, in 2 different geographical regions, with databases replicated from 1 site to the other.

Estimate your storage needs for the next 5 years, and even more

Are you able to do that? It’s not that simple. Most of the time you can estimate for the next 2-3 years, but more than that is highly uncertain. Maybe a new project will start and will require much more storage? Maybe you will have to provision more databases for testing purpose? Maybe your main software will leave Oracle to go to MS SQL or PostgreSQL in 2 years? Maybe a new CTO will arrive and decide that Oracle is too expensive and will build a plan to get rid of Oracle. We never know what’s going to happen in such a long time. But at least you can provide an estimation with all the information you have now and your own margin.

Which margin should I choose?

You probably plan to monitor the free space on your ODA. Based on the classic threshold, higher than 85% of disk usage is something you should not reach. Because you may not have a solution for expanding storage. 75% is on my opinion a good space usage you shouldn’t reach on ODA. So consider 25% less usable space than available when you do your calculations.

Get bigger to last longer

I don’t like wasting money or resources for things that don’t need to, but in that particular case, I mean on ODA, after years working on X3-2, X4-2, and newer versions, I strongly advise to choose the maximum number of extensions you could. Maybe not 76TB on an ODA X8-2M if you only need 10TB, but 50TB is definitely more secure for 5 years and more. Buying new extensions could be challenging after 3 or 4 years, because you have no guarantee that these extensions will still be available. You can live with memory or CPU contentions, but without enough disk space, it’s much more difficult. Order your ODA fully loaded to make sure no extension will be needed.

The more disk you get, the more fast and secure you are

Last but not least, having more disks on your ODA maximize the throughput: because ASM is mirroring and stripping blocks on all the disks. For sure on NVMe disks you probably won’t use all that bandwidth. More disks also adds more security for your data. Loosing one disk in a 4-disk ODA requires the rebalancing of 25% of your data to the 3 safe disks, and rebalancing is not immediate. Loosing one disk in a 8-disk ODA requires the rebalancing of much less data, actually 12.5% assuming you have the same amount of data on the 2 configurations.

A simple example

You need a single-node ODA with expandable storage. So ODA X8-2M seems fine.

You have an overview of your databases growth trend and plan to double the size in 5 years. Starting from 6TB, you plan to reach 12TB at a maximum. As you are aware of the threshold you shouldn’t reach, you know that you’ll need 16TB of usable space for DATA (maximum of 75% of disk space used). You want to make sure to have enough FRA so you plan to set DATA/RECO ratio to 80%/20%. Your RECO should be set to 4TB. Your ODA disk configuration should have at least 20TB of usable disk space. A 8-disk ODA is 19.8TB of usable space, not enough. A 10-disk ODA is 24.7TB of usable space for 19.76TB of DATA and 4.94TB of RECO, 23% more than needed, a comfortable additional margin. And don’t hesitate to take a 12-disk ODA (1 more extension) if you want to secure your choice and be ready for unplanned changes.

Conclusion

Storage on ODA is quite expensive, but don’t forget that you may not find a solution for an ODA with insufficient storage. Take the time to make your calculation, keep a strong margin, and think long-term. Being long-term is definitely the purpose of an ODA.