Network configurations on ODA

Introduction

ODA is an appliance, that means that everything is included in a box, and this box could eventually run its stuff only being connected to power. But it would be quite useless as nobody could connect to the databases. ODA needs network links to deliver the service.

Starting from ODA X8, there are multiple possible configurations regarding network, but also limits to what you can do. Let’s have a glance at the possibilities.

Network cards on a single-node ODA

A minimal configuration will be composed of 2 network cards.

One is for server management, also called the ILOM. ILOM is something like a server in the server itself. It consists of a small-footprint machine dedicated to manage the hardware side of the ODA. When the ODA gets powered, this ILOM is available a few minutes after you plug in the server, even if you don’t actually power up the server itself by pressing the button. There is only one physical port for this network card, it’s even not a card as it’s soldered on the server and you cannot remove it. This physical port is an Ethernet port limited to 1Gbps, so you will not be able to connect it to faster networks. It’s not a problem because you’ll never need more. There is no indication if this port could eventually use lower speeds, like 100Mbps, but using 100Mbps in 2021 would be quite odd.

The other network card is a real one, connected to the PCIe bus slot 7. You can choose between a quad-port Ethernet card, or a dual-port SFP28 card.

Ethernet card is for copper network with RJ45 cables and plugs, and is compatible with 10Gbps switches.

SFP28 card is for fiber network with optical fibers and gbics connectors, and is compatible with 10Gbps or 25Gbps switches.

2 more cards are optional. Like the previous one, you can choose between quad-port Ethernet or dual-port SFP. They can be useful for those planning to have more network interfaces for various purposes. These cards will be available in port 8 and 10.

So, when you order your ODA, you’ll have to choose which type of card for the first one, and if you want additional ones (1 or 2) and which kind of additional cards. Make sure to order exactly what you need.

Network cards on a HA ODA

HA ODA is basically 2 single-node ODAs without DATA disks inside. Disks are deported through a SAS link to one or two storage enclosures, meaning that storage capacity is bigger, but unfortunately without the speed of NVMe disks.

As one HA is 2 single-node ODAs, you’ll find 2 ILOM cards (1 per server) and 2 cards as a default (in slot 7 of each server). Each node can only have 1 additional card into slot 10. It’s not possible to add more cards for your own networks, because the other slots are dedicated to HBA SAS controllers (2x) and a specific network card.

As you may know, one of the benefit of having a HA ODA is High Availability itself, thanks to Real Application Cluster technology. This technology rely on shared storage but also on high bandwidth private network (interconnect) to link both nodes together. This private network has a dedicated card on each node on slot 1, and this network is really private because it never goes out of your ODA. It makes use of a SFP28 card of the same type as the others. But dedicated cables are provided to make a direct link between node 0 and node 1. It means 2 things: both nodes need to be very near, and you do not have to care about this network configuration: no ports to dedicate on your switch, no IP configuration to provide, etc.

As a result, network configuration on a HA ODA is not very different than network configuration of 2 single-node ODAs, but one of the additional card is mandatory and dedicated to interconnect only.

ILOM network configuration

ILOM network configuration will be composed of an IP address, a netmask and a gateway for being available from other networks. You will mainly connect to this network with a web browser on https port, or eventually with SSH terminal. ILOM is used for initial deployment or reimaging of the server, and also for diagnostic purpose.

Public network configuration

Public network is dedicated to connect your applications or clients to the databases. A public network will be based on 2 bonded interfaces on the same network card. If you ordered your ODA with 1 Ethernet card, you can choose between btbond1 or btbond2, btbond1 being a bond of the first 2 ports, btbond2 being a bond of the last 2 ports.

If you ordered your ODA with a SFP card, there is only one bond possible btbond1 on the 2 ports of the card.

For this network you will need an IP address, a netmask, a gateway and also DNS and NTP information because your ODA will need to resolve names and update its clock.

Other networks configuration (user defined networks)

You can configure other networks for additional links depending on your needs and how many cards you chose. For single-node ODAs, up to 6 networks are possible when using Ethernet (public + 5 user-defined networks) and up to 3 using SFP (1 public and 2 user-defined). For HA ODAs, up to 4 networks are possible when using Ethernet and up to 2 using SFP. These networks will be used for backup purpose, in order to dedicate a link to backup stream, administrative purpose in case you have a dedicated network for administrators, or whatever you need.

Each network you will create will be based on 2 bonded ports of a network card.

Note that bond’s names are fixed and follow this mechanism:

• 1st card has btbond1 and eventually btbond2 if it’s an Ethernet card
• 2nd card has btbond3 and eventually btbond4 if it’s an Ethernet card
• for single-node ODAs, 3rd card has btbond5 and eventually btbond6 if it’s an Ethernet card

For each of these user-defined networks you will need an IP address, a netmask, a gateway and a optional VLAN tagging if needed.

To create a user-defined network, use odacli create-network and provide the physical bond, the IP configuration, for example you could create a backup network on the second bond of your primary Ethernet card with:

odacli create-network -w Backup -m Backup -s 255.255.255.0 -p 172.30.52.4 -t BOND -n btbond2

You can also display your actual networks with:

odacli list-networks

Limits

Configuration is quite flexible, but there are some limits.

Manual configuration of these interfaces is not supported. Everything is either done when deploying the appliance (public network configuration) or with odacli (additional networks configuration).

You can tweak your network scripts, but it may prevent you from applying a patch or using some odacli functions. So I do not recommend to do that.

LACP is not supported, networks are always composed of 2 bonded ports, but it’s only in active/backup mode. You cannot aggregate networks interfaces to benefit for double the speed of one. If you need more speed, connect your ODA to 10Gbps Ethernet or 25Gbps SFP switches to maximize throughput if you really need such bandwidth. Aggregating “slow” 1Gbps links makes no sense. Active/backup is enough for most cases.

Jumbo frames are not proposed during network configuration, if you really need them, for example for a backup network, you can add the settings to your network interfaces configuration files but don’t forget to remove them before patching or modifying networks through odacli.

echo 'MTU="9000"' >> /etc/sysconfig/network-scripts/ifcfg-p2p1
echo 'MTU="9000"' >> /etc/sysconfig/network-scripts/ifcfg-p2p2
echo 'MTU="9000"' >> /etc/sysconfig/network-scripts/ifcfg-btbond3
ifdown btbond3; ifup btbond3

If you still use an old 1Gbps fiber switch, the ODA will not manage to connect to the network, even if the network link is detected. 1Gbps SFP makes no sense compared to 1Gbps Ethernet.

Using both Ethernet and SFP is not possible, and it’s probably for now the main drawback regarding the network on ODA. Mixing up the 2 types of network cards is possible, but you will not be able to use both of them. An ODA with the 2 types of card only makes sense when you plan an upgrade to SFP later (with a reimage) or if your datacenters don’t support the same kind of networks and you want your ODAs to be identical. This is probably a good idea to have the same cards on each ODA.

It’s not possible to make a bond between different physical cards. It would make sense as the card itself could have a failure, but there is no way for such configuration with odacli. Don’t forget that a real Disaster Recovery solution is the only answer for extended hardware troubles today.

Finally, if you decided to go for SFP, don’t forget to order the gbics corresponding to your configuration, they are not provided as a standard on ODA, they are optional. Make sure to have the correct gbics according to the network specification of the cards and your switch on both sides of the fiber cable. I heard about some incompatibilities between network card, gbics plugged into the card, gbic plugged into the switch and the switch itself. It’s not an ODA specific subject but make sure that this point will not delay your ODA project.

ODA and VLANs

VLAN tagging is fully supported on ODA, but I recommend to use transparent VLANs and tag the switch port instead of configuring this tagging on the ODA itself. You can still create additional VLANs on top of the public interface, but you will share the bonding for multiple purposes, not the best option. One bonding for one VLAN is probably a better configuration.

Conclusion

There is a lot of possibilities regarding network configuration on ODA, but as usual, take the time to define what you need, look at your available ports and speeds with your network administrator, and prepare carefully information like IPs and hostnames. Changing something on the network after deployment is not recommended and even not always possible.