Broadcast Domains and IP Spaces in NetApp Clustered Data ONTAP 8.3

By on 06/02/2015.

Clustered Data ONTAP (cDOT) is NetApp’s newest storage operating system. Essentially, it takes all the goodness of ONTAP operating in 7-Mode and tethers it together into a clustered environment. This allows seamless scalability, non-disruptive upgrades of software and hardware and flexibility in storage provisioning.

One area of IT where this is especially useful is in a multi-tenant environment. With cDOT, storage administrators can segment storage to users much like virtual machines in an ESX farm. You share the same physical resources, but you abstract the storage layer via virtualized storage instances known as Storage Virtual Machines (SVMs).

Prior to cDOT 8.3, providing true multi-tenancy had some limitations.

  • Each LIF had to have a unique IP address in the cluster.
  • In some cases, LIFs could end up on ports that were not on the name network as the LIF, even with failover groups properly defined (such as mistakenly migrating a LIF manually).
  • LIFs could end up on ports that had different MTU sizes than their original source ports.

cDOT 8.3 solved those problems with the addition of Broadcast Domains and IP spaces.

Broadcast Domains and…. Roomba?

The easiest way to think of a broadcast domain is to think of a Roomba.

Roombas are cute little robots that can be programmed to “know” your room. They sense walls, furniture and other obstacles to avoid as they clean. If you wanted your Roomba to clean only your bedroom, you would stick the Roomba in there and shut the door. The walls and door would keep the Roomba confined to where it belongs.

However, if you left the door open, the Roomba could eventually find its way out and start cleaning the rest of your house. If one room had, say, super high shag carpeting, the Roomba might get stuck and you get a ROOMBA OUTAGE!

So how does this relate to broadcast domains?

Well, broadcast domains are the bedroom. The data LIF is the Roomba.

Broadcast domains are essentially a walled off area for data LIFs. They are a logical division of computer networks in which all nodes can reach each other by broadcasting at the data link layer (layer 2).

broadcast-domain

When you build a broadcast domain in cDOT, you populate it with physical ports that are cabled to a switch on the same subnet. Then you create your logical interfaces (LIFs) on the physical ports. Now, a data LIF can only migrate within that broadcast domain, even if you try to do it manually.

This is an example of two broadcast domains (BD-100 and BD-200) with two separate VLANs. None of the ports in VLAN 100 can host data LIFs specified for ports on VLAN 200. Broadcast domains should all follow layer 2 network logic – anything on the same subnet that doesn’t need a router to talk to other things should live in the same broadcast domain.

broadcast-vlan

When would I use a custom broadcast domain?

There is already a default broadcast domain in cDOT, appropriately named “Default.” But when would you need a user-created broadcast domain?

In all cases, you should use them for networks that are already segmented on a layer 2 network via VLAN or physical cabling. One use case for using broadcast domains would be separating out data networks running on 10GB ports and management networks running on 100MB or 1GB ports.

This would only be done when the networks are on different subnets.

Naturally, you wouldn’t want data traffic to end up on the slower links, and you probably don’t want all that management traffic (which could include OnCommand ZAPIs, SNMP, SSH, etc) to interfere with data traffic. With a broadcast domain, you can guarantee that those networks will remain separate. However, if they are on the same subnet, you will want to use a failover group instead.

In the following example, I created a management broadcast domain to go along with my main “Default” broadcast domain:

cluster::> broadcast-domain show
 (network port broadcast-domain show)
IPspace Broadcast                                        Update
Name    Domain Name MTU    Port List                     Status Details
------- ----------- ------ ----------------------------- --------------
Cluster Cluster     1500
                           parisi-fs-01:e0a              complete
                           parisi-fs-02:e0a              complete
Default Default     1500
                           parisi-fs-01:e0c              complete
                           parisi-fs-01:e0d              complete
                           parisi-fs-02:e0c              complete
                           parisi-fs-02:e0d              complete
MGMT                1500
                           - -
3 entries were displayed.

All of my management LIFs live on port e0c:

cluster::> net int show -role node-mgmt,cluster-mgmt
 (network interface show)
            Logical       Status     Network            Current       Current Is
Vserver     Interface     Admin/Oper Address/Mask       Node          Port    Home
----------- ------------- ---------- ------------------ ------------- ------- ----
cluster
            cluster_mgmt  up/up      10.63.21.16/18      parisi-fs-02 e0c     true
            mgmt1_node1   up/up      10.63.22.164/18     parisi-fs-01 e0c     true
            mgmt1_node2   up/up      10.63.23.186/18     parisi-fs-02 e0c     true
3 entries were displayed.

And my failover groups all include e0c in the eligible ports for failover targets:

cluster::> failover-groups show
 (network interface failover-groups show)
                 Failover
Vserver          Group            Targets
---------------- ---------------- --------------------------------------------
cluster
                 Default
                                  parisi-fs-02:e0c, parisi-fs-02:e0d,
                                  parisi-fs-01:e0c, parisi-fs-01:e0d

If those ports were 100MB (and on different subnets), I’d want to segment them off into their own broadcast domain.

To do this, no data LIFs can live on the port.

cluster::> broadcast-domain remove-ports -broadcast-domain Default -ports parisi-fs-01:e0c
 (network port broadcast-domain remove-ports)

Error: command failed: Port "parisi-fs-01:e0c" cannot be used because it is currently the home port - or current port of a LIF.

So I’d need to move the LIF and ensure it’s on a new “home” port first.

cluster::> net int migrate -vserver cluster -lif mgmt1_node1 -destination-node parisi-fs-01 -destination-port e0d
 (network interface migrate)

cluster::> net int modify -vserver cluster -lif mgmt1_node1 -home-node parisi-fs-01 -home-port e0d
 (network interface modify)

Then I can remove it:

cluster::> broadcast-domain remove-ports -broadcast-domain Default -ports parisi-fs-01:e0c
 (network port broadcast-domain remove-ports)
cluster::> broadcast-domain show
 (network port broadcast-domain show)
IPspace Broadcast                                        Update
Name    Domain Name MTU    Port List                     Status Details
------- ----------- ------ ----------------------------- --------------
Default Default     1500
                           parisi-fs-01:e0d              complete
                           parisi-fs-02:e0c              complete
                           parisi-fs-02:e0d              complete
MGMT                1500
                           - -

Then I can stick e0c on node1 into my new broadcast domain and move the management LIF back.

cluster::> broadcast-domain add-ports -broadcast-domain MGMT -ports parisi-fs-01:e0c
 (network port broadcast-domain add-ports)
cluster::> broadcast-domain show
 (network port broadcast-domain show)
IPspace Broadcast                                        Update
Name    Domain Name MTU    Port List                     Status Details
------- ----------- ------ ----------------------------- --------------
Default Default     1500
                           parisi-fs-01:e0d              complete
                           parisi-fs-02:e0c              complete
                           parisi-fs-02:e0d              complete
MGMT                1500
                           parisi-fs-01:e0c              complete

Note how my failover groups need to be modified, though. I now have nowhere to fail that LIF to!

cluster::> net int modify -vserver cluster -lif mgmt1_node1 -home-node parisi-fs-01 -home-port e0c
(network interface modify)

Warning: Changing the home port to a new broadcast domain may result in a momentary loss of connectivity while the LIF is moved to the new port.
Do you want to continue? {y|n}: y

Warning: The configured failover-group has no valid failover targets for the LIF's failover-policy. To view the failover targets for a LIF, use the "network interface show -failover" command.

cluster::> failover-groups show -vserver cluster -failover-group MGMT
(network interface failover-groups show)

Vserver Name: cluster
Failover Group Name: MGMT
Failover Targets: parisi-fs-01:e0c
Broadcast Domain: MGMT

Notice also that the original failover group the port was in no longer has that port as a member:

cluster::> failover-groups show -vserver cluster -failover-group Default
(network interface failover-groups show)

Vserver Name: cluster
Failover Group Name: Default
Failover Targets: parisi-fs-02:e0c, parisi-fs-02:e0d, parisi-fs-01:e0d
Broadcast Domain: Default

And I can’t migrate the LIF now (by design):

cluster::> net int migrate -vserver cluster -lif mgmt1_node1 -destination-node parisi-fs-01 -destination-port e0d
(network interface migrate)

Error: command failed: Failed to migrate LIF "mgmt1_node1" on Vserver "cluster". Destination port "parisi-fs-01:e0d" is in a different broadcast domain than the port currently hosting the LIF.
Use the "network port broadcast-domain show" command to view port broadcast domains.

Segmenting LIFs on same subnet? Use failover groups.

In the example above, we used a broadcast domain to conduct the logical separation of management and data networks which are on different subnets. However, if they are on the same subnet, they’d be in the same broadcast domain. In that case, use failover groups.

Failover group vs. Broadcast domain?

In cDOT 8.2.x and prior, LIF migrations during failover events were only covered by failover groups. There were a couple issues with this approach.

  • For one, failover groups required manual configuration and were often forgotten – that is, until something bad happened.
  • Two, when something bad happened, LIFs could potentially fail over to ports that were either configured for different subnets, VLANs or had different throughput configurations (including different MTU sizes).
  • Three, even if a failover group was configured, a storage administrator could manually migrate a data LIF to a port that was invalid, unbeknownst to them.

In cDOT 8.3, broadcast domains prevent incorrect failover of data LIFs. They are still manually configured, but ports can now be segmented off into more logical networking areas. Failover groups still exist, however, so you can set up granular network failover logic within your broadcast domains.

Upgrading from cDOT 8.2.x to 8.3

What happens to failover groups when you upgrade?

Since failover groups have the same intent and purpose as broadcast domains (albeit not as effectively), when you upgrade, failover groups get converted into broadcast domains, provided the ports all share similar configurations.

cDOT protects you on upgrades from having ports that are mismatched in configuration with warning messages, such as when MTU sizes are different on ports in the same failover group. Additionally, the failover groups get added to the broadcast domains. Storage admins can later add/remove ports as needed.

Per-node failover groups

In cDOT 8.2.x and prior, failover groups were also created for node-specific LIFs, such as the cluster LIFs, node management, etc. However, there is no more concept of “node-specific” LIFs in cDOT 8.3 – everything that was in a node SVM is now considered a part of the cluster SVM.

As a result, any node-specific failover groups/broadcast domains that get created and carried over can be deleted after upgrade, once ports have been confirmed to be in the appropriate cluster SVM failover groups.

IP Spaces

Another new feature added to cDOT 8.3 was IP spaces. This allows a single cluster to leverage IP addresses used by clients from multiple disconnected networks.

Expanding on the Roomba example, this would be essentially like being able to now buy multiple red Roombas for your house for multiple rooms without anyone getting confused about which Roomba was which.

Or, for a more apt comparison, it’s like being able to have multiple home wireless routers using the same 192.168.1.x network without worrying about duplicate IP addresses. And unlike ONTAP operating in 7-Mode, these IP spaces don’t share the same routing table (bye bye vfiler0!). Instead, each SVM has its own routing table for a true multi-tenant storage environments!

TR-4182: Ethernet Storage Design Considerations and Best Practices covers these in detail.

TR-4160: Secure Multi-Tenancy in cDOT covers the notion of multi-tenant design in detail.

Justin Parisi
Tech Mktg Engineer at NetApp
Justin is a Tech Marketing Engineer for all-things NFS around Data ONTAP at NetApp. He is a VMware vExpert, Cisco Champion, and a member of the NetApp A-Team. He also enjoys comic books, video games, photography, music, film, and current events/politics.

5 Comments