Infrastructure at your Service

Marc Wagner

ODA, network interface and configure-firstnet

Deploying new Bare Metal ODA X7-2S or 2M, I have been curious how configure-firstnet would interact with the fiber and copper ethernet network interfaces. Reading documentation on the web, I could not clearly understand if it is mandatory to have the ODA connected to the LAN when performing an ODA reimage and/or running the configure-firstnet in additionnal of having ILOM connection. After digging deeper and few tests, I wanted to share my experience in this blog.

configure-firstnet

Running configure-firstnet might need a little attention as it can be run just one time. In the example below, I’m running the firstnet configuration with a VLAN.

configure first net

 network-script files

The network script files are stored in /etc/sysconfig/network-scripts.

Bounding is configured on btbond1 interface as :

  1. em2 as primary interface.
  2. active-backup mode : em3 is used as backup only and will be used if em2 is failling
    BOUNDING_OPTS=”mode=active-backup miimon=100 primary=em2″

The only officially supported bounding option on the ODA is “active-backup”. Manually updating the ifcfg-btbond1 file with BOUNDING_OPTS=”mode=4 miimon=100 lacp_rate=1″ in order to implement LACP (Link Aggregation Control Protocol) would work if your switch is configured so, but is not supported. Recommendation would be to use ODA with “active-backup” mode only.

This can be seen in the ifcfg-btbond1 configuration file.

10.80.6.17-002-btbond1

After running configure-firstnet, using a VLN, a new file ifcfg-btbond1.<vln_id> will be created. This file is having all the IP configuration (IP address, netmask, gateway). If no VLAN is used, the IP configuration will be added into the ifcfg-btbond1 configuration file.

10.80.6.17-004-vlan

If we look more closely to the network configuration file, we will see em2 and em3 been configured with btbond1 as master.

10.80.6.17-001-em2&3

As we can see here there is, so far, no reference in the network configuration file as if you are using fiber or copper ethernet interface.

em2 and em3 interface : Fiber or copper?

em2 and em3 interfaces are automatically connected either on fiber or on copper ethernet according to the physical connection. Em2 and em3 interfaces would then either be using the 2 fiber ports (SFP28) or the 2 ethernet ports (NET1 – NET2).
In fact, as soon as a GBIC converter is plugged into the fiber channels and a reboot is performed, the em2 and em3 will be automatically link to the fiber adapter. And no need to have any fiber cabling.

No GBIC converter installed on the ODA

IMG_1158 (Small)

em2 and em3 interfaces would be seen as Twisted Pair.

10.80.6.17-003-no gbic

GBIC converter

IMG_1160 (Small)

GBIC converter installed on the ODA

IMG_1161 (Small)

After a server reboot, em2 and em3 interfaces would be seen as Fiber.

10.80.6.17-004 (after plugging gbic and reboot)

Conclusion

Based on this short experience, we can see that the Fiber is detected as soon as a GBIC adapter is plugged into the SFP28 interfaces. The ifconfig network scripts are totally independent of this choice. Therefore, there is no need to have the ODA em2 and em3 network connections to reimage and run the configure-firstnet. These connections will be mandatory for the next step when we will create the appliance.

15 Comments

  • Mark says:

    Hi Marc,

    I am facing a request that a client would want to switch from RJ-45 to the SFP interfaces on the X7-2s .
    Would the same principal apply ?
    Connect the SFP ports and disconnect the RJ-45 ports and on reboot the bond0 would connect ot the em2/3 interfaces still ?

    Nice article btw

    grtz Mark

    • Marc Wagner says:

      Hi Mark,
      Thanks for your feedback. Exactly. You will only have to unplug the RJ45 cable on EM2 and EM3 interfaces, plug your GBIC for fiber and reboot.
      Cheers,
      Marc

  • JOSE NETO says:

    Hi Mark

    I deployed ODA x7-2 HA virtualized with ISO 18.3 using RJ-45 ethernet copper cable in the 1GB network, we already have it configured and with VMs created, everything is ok. But we want to switch to using fiber optics on the SFP28 ports.

    By its explanation above this modification would be automatic, it would only have to put the fiber and restart the node, with that the eth2 and eth3 would already work via fiber

    Is my understanding correct?

    There is a command in Dom0 that says the following:
    oakcli [configure][network][-publicNet]
    The command is used to configure network after swapping the public interface from copper to fiber or vice versa

    “oakcli configure network -publicNet” can be used for this situation?

    • Marc Wagner says:

      Hi Jose,
      The tests I have been able to perform here were on a bare metal installation. The virtualized architecture is quite different and much more complex. I don’t have the hindsight of changing public network from copper to fiber on a virtualized architecture. I would recommend you to check if the oakcli is working as expected if the Appliance is already deployed and to perform some test in a lab first. Otherwise oracle support might be a good choice to validate your method of procedure. On a bare metal installation, it is quite more simple and the changes will be seen after a simple reboot.
      Good luck.
      Cheers,
      Marc

  • Noor says:

    Hi Marc,
    I have a request to move the X7-2-HA ODA (Bare metal config) to another network configuration. The ODA is running live production databases with Dataguard. The move would require a change in the public and virutal ip’s. How can we update the network configuration without rebuilding the ODA again? Is there an easy way?
    I see that on the oda-admin page there is a possibility of updating the ip address but i think that will cause a problem with the cluster?
    Regards,
    Noorul

    • Marc Wagner says:

      Hi Noorul,
      The configure-firstnet can only be executed one time, and oracle documentation is clear about this. You could update the linux network-scripts configuration, it is a linux system but this is highly not recommended and this might bring many problem on next patching. And certainly others… As per my knowledge, the only solution for you would be to reimage the ODA.
      I would encourage you to open a SR by Oracle and ask for a confirmation. It would be nice to share then the outcome of your discussion with oracle support here as well.
      Thanks.
      Good luck.
      Regards,
      Marc

  • Noorul says:

    Hi Marc,
    Thanks for your reply. I have checked with Oracle and the following is the document that deals with the network changes for the old oak stack.
    ODA: How to Change Oracle Database Appliance IP Addresses Post Deployment (Doc ID 1504734.1)

    For the newer dcs stack or X7 ODA’s the only solution is to re-image the ODA unfortunately.

    Regards,
    Noorul

  • Jefferson Rodrigues says:

    Hi Marc,

    We bought a Server-ODA-X7HA and it is installed.

    We have two networks and two domains and one scan configured for each domain.

    The ode when it receives request from the 192.168.X.X network is delivered by the 10.200.x.x network and asymmetric routing occurs.

    I’m in touch with the support and nobody can help.

    can you help me?

    • Marc Wagner says:

      Hi Rodrigues,
      Hard to make any troubleshooting with so poor information. Not sure if you can but please provide your ODA IP configuration and provide a clear output of the issue.
      Thanks.
      Regards,
      Marc

  • Jamsher says:

    Hi Mark,

    As a part of infrastructure update We are also planning to upgrade our ODA X7-2M Server from Copper Cable RJ45 to Fiber Cable SFP+
    I already open SR with Oracle support for this change.
    They are asking to upgrade firware
    Upgrading the Onboard SFP28 Ports on ODA X7-2 Server Node (Doc ID 2373070.1)
    Can you please confirm if its require. or only removing RJ45 and plogging in SPF+ Module is enough

    Best Regards
    Jamsher

  • Jamsher says:

    Hi Marc,

    Thanks for confirmation. Really Appreciated. Now we are waiting for SFP Module to deliver so we can make this network upgrade.

    Thank again.

    Best Regards
    Jamsher

  • ilyes says:

    Hello Marc,
    We deploy x8-2M that have oracle dual port 25gb ethernet with SFP Copper 1G. installed as bonding
    questions:
    Can we disable bonding and use each port for separate network?
    How can we configure em1 because we dont see this interface management console ?

    thanks a lot

    • Marc Wagner says:

      Hi Ilyes,
      Yes, you can decide not to use bonding. But this should be already selected when you run first time the configure-firstnet command by replying “no” to the question “Using bonding public interface”. I personally always use bonding. Best practice if a link is down the second one will be available. ODA are working in active backup mode. I would prefer to add other network cards to use other networks.
      The non-bonded configuration enables you to create a physical network on the unused physical interface. When you use the command configure first-net to plumb the network as a non-bonded interface, there are then two physical network interfaces, em2 and em3 before X8 and p7p1 and p7p1 with X8. One of the interfaces is for the public network, the second interface is available for you to create another network.
      As per my knowledge em1 can not be used, reserved for Oracle.
      Regards,
      Marc

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Marc Wagner
Marc Wagner

Consultant