In Affinity Networking, Enterprise Data Center, Featured, Industry Insights, Network Evolution, SDN

A little while ago I wrote about the differences between routing and switching, or probably more the difference between ethernet and IP forwarding. Then focus of that article was very much on the differences between the two from a forwarding hardware perspective. This article last week from Brent Salisbury triggered a bunch of additional thoughts around scale and size.

As some sense of disclosure, in my previous job at Nortel/Avaya I was part of the team that pushed Shortest Path Bridging (SPB) spearheaded by Paul Unbehagen. I am a fan of SPB, I believe it is an extremely well thought out mechanism to provide network based virtualized ethernet networks with many real life practical roots in the learnings from MPLS and early more static solutions like PBB and PBT.

One of the questions I have been asked many times with respect to SPB is: "how big can I scale my SPB managed VLAN?"  That question should also be asked for TRILL or even VXLAN or NVGRE overlay L2 solutions provided by NSX or anyone else. And the answer to the question should be the same for all of them.

Whatever the means of virtualizing an ethernet network, the two most basic characteristics of an ethernet LAN stay the same. It is a single broadcast domain and it needs to be loop free. It does not matter whether this is a domain designated by an ISID and managed by SPB, designated by a VNI and managed by a tunnel controller, or whether this a good ole VLAN as we know it today. They all behave the same. A broadcast will reach any device in the domain. The switches achieve this by sending this packet out every port along a spanning tree that covers every (edge) switch in the network that is part of that virtual network. Some by making use of underlying multicast capabilities, others by replicating packets to all edge switches that need it (worthy of a discussion all by itself).

And that realization leads to my answer to the scale question. Don't create a L2 domain in a virtualized environment any larger than you would in a non virtualized environment. There is no hard number to give, I have seen VLANs with more than 2000 devices work flawlessly, others with just a few 100 getting pummeled with broadcasts. The size should be the same as if you were to build a single VLAN ethernet network. You know your applications, devices and traffic patterns best, don't change what you have done simply because you can virtualize it.

What modern day L2 virtualization gives you besides a way around 4000 VLANs, is a convenient way to spread Virtualized LANs across a large network. Where previously you had to manually create a tree that connected all portions of a VLAN together and run a flavor of STP to ensure you removed the loops you intentionally created for redundancy, todays mechanisms take that into the 21st century using ISIS for TRILL and SPB for loop free network reachability and some additional smarts to ensure you do now have to manually "connect" all endpoints of a virtualized LAN. The protocols take care of exchanging virtual network IDs for you. And while they make it easier to create larger individual virtual networks, don't be tempted. Those same broadcast, flooding and external loop concerns are still there.

Here at Plexxi we strongly believe in a next step of evolution, with a centralized controller in charge of physical and logical network topologies and forwarding behavior. We believe that a controller with network wide views, policies and control can make better decisions that mix network behavior, physical topology and global policies into actual packet forwarding. That architecture holds true for virtualized L2 networks too. A central controller that maintains loop free topologies and is also responsible for virtual network membership (which MAC on which port belongs to which virtual network), performs the same function as ISIS does for SPB or TRILL. 

Brent mentions two other key functions of the overall solution that are significantly different between more traditional network based virtualization and server based virtualization NSX, Plumgrid, Microsoft and many others provide: MAC Learning and encapsulation. This is where controller based architectures and VXLAN (and NVGRE) encapsulation take a significant step forward. And these two differences can create some really cool solutions, especially if you start thinking of the hybrid solutions that can be created when you take the best of both worlds… 

To read posts on similar topics, check out:

Showing 2 comments
  • Colin Dixon
    Reply

    Just a minor note. You don’t have to keep the topology loop free. You just need to keep any given route, especially whatever routes you use for broadcast, loop free. This gives you a lot more freedom than when you must keep your usable part of the topology loop free.

    To illustrate this see the PAST paper that some of us at IBM research wrote: http://conferences.sigcomm.org/co-next/2012/eproceedings/conext/p49.pdf

    • Marten Terpstra
      Reply

      Colin, apologies for my liberal use of “topology”. Separating unicast from multicast/broadcast forwarding absolutely creates better degrees of freedom (Plexxi forwarding topologies are different for unicast and multicast/broadcast today). But while that freedom allows for separation of unicast and multi/broadcast (and thus perhaps reduce the interference between the two) in the fabric, leafs of that specific domain will still feel that potential pain. Perhaps I did not clearly articulate that the size of a L2 domain is not a function of the (type of) fabric, but a function of the amount of leafs/end devices, regardless of the fabric technology.

Leave a Comment

Contact Us

We're not around right now. But you can send us an email and we'll get back to you, asap.

Start typing and press Enter to search