Data centers: Cisco's multi-tier architecture vs. the Arista 2-tier cloud network

BradReese.Com Blog by Brad Reese
Archive

Aironet

Competitive Lab Tests

Subscribe to the BradReese.Com Blog

Data centers: Cisco's multi-tier architecture vs. the Arista 2-tier cloud network
Fri, 04/23/10 - 3:27pm Add your comment

Earlier in the week I blogged that choosing Arista Networks over Cisco, could save data centers $13.3 million.

Breaking News: Arista Wins Best of Interop Las Vegas 2010

Tellingly, networking authority - Nick Lippis, has recorded a podcast about the new Arista 7500 modular 10Gb ethernet switching platform.

Also interestingly, Arista features on its website the following whitepaper:

Cloud Networking Designs by Lippis

In the above whitepaper, Lippis notes, "Take server density on a scale we have not seen previously and add ultra application demand at load and you have the requirements for a new kind of networking."

And what's this new kind of networking you may ask?

Well, Lippis expounded further, "To accommodate these requirements a two-tier network model is being considered consisting of what I call a 'Cloud Access' tier and 'Cloud Core' tier. The Cloud Access tier connects servers while the Cloud Core consists of a series of non-blocking switches delivering mesh connectivity between non-blocking Cloud Access switches. The Cloud Core also connects storage and wide area services/routers to the cloud. Within both cloud tiers are switches that provide layer 2 and layer 3 services giving the cloud architect design options of deploying all layer 2, all layer 3 or a hybrid yielding choice as to where to place the layer 2/layer 3 boundary."

Lippis also revealed, "Only two firms, Cisco and Arista have developed new operating systems and hardware for this market. While Cisco does not tout a two-tier architecture in its Data Center 3.0 program, its Nexus data center switches can clearly be configured in this form. For example, its high end Nexus 7000 would occupy the Cloud Core while its Nexus 5000/2000 occupies the Cloud Access tier. The Nexus 2000 provides GbE connections to servers while obtaining configuration and NX-OS services from the Nexus 5000 via 10GbE placed in end-of-row. The Nexus 2000 and 5000 may be two separate physical devices but they are logically one, making up the Cloud Access tier. In this scenario the Nexus 2000 is a line extender and I expect to see others introduce a similar approach as it delivers the cabling efficiency of top-of-rack and network management operational efficiency of end-of-row. The layer 2/layer 3 boundary resides in the Nexus 7000."

Finally Lippis concluded, "The two and three-tier models will co-exist with three-tier being the network architecture in building/campus networks and non-cloud/high performance data centers. But for the high-end cloud and high performance data centers, the two-tier model offers the attributes of low latency, cost and packet throughput required."

Below, visually view Cisco's multi-tier architecture vs. the Arista 2-tier cloud network

According to Arista Networks, taking both vendors best practices recommendations yields a network that looks like this:

In Figure-1 below, Cisco on the left uses the Nexus 1000v to provide Virtual Machine Policy, The Nexus 2000 for cost-effective GbE and 10GbE MUXing into the Nexus 5000 which makes a Layer-2 forwarding decision to the Nexus 7000 which handles L3 Routing, Address Resolution, Access Control, etc.

Arista combines many of these capabilities into a smaller number of layers- The Arista 7100 provides Layer-2, Layer-3 and some VM services. The Arista 7500 provides a high-throughput low-latency core interconnect for the Arista 7100s in the 'leaf' or access layer.

Figure-1:

The Figure-2 diagram shows the path packets from one server on one VLAN must take to get to another server on another VLAN, even if the servers are sitting right next to each other in the cabinet. (Note: The Nexus 1000v is only necessary in a VMware virtualized environment).

The Arista 2-Tier design can make the intelligent L3 forwarding decision on the first hop, without going through MUX's and Layer-2 only devices.

Given the published latencies of these products, the Cisco network architecture can deliver a port-to-port L3 forwarding latency of 94-154 microseconds for max frame sizes and the data will be susceptible to congestion and oversubscription by 4:1 on the Nexus 2000 uplinks, 5:1 on the Nexus 5000 uplinks, and 4:1 again on the Nexus 7000 because of linecard oversubscription.

The Arista Network design forwards the same frame sizes at 600 nanoseconds, approximately 256 times faster, and has wirespeed forwarding for this example. If the network engineer designs in oversubscription by using a smaller number of uplinks than host-facing links then the Arista Network can be designed to have network-based oversubscription.

Figure-2:

Similar to above, in Figure-3 the packets from one host to another, in the same cabinet, in the same VLAN must go through the Nexus 2000 --> Nexus 5000 --> Nexus 2000 to make any L2 forwarding decision.

The Arista products make the L2 forwarding decision on the first ingress port.

Given the bulk of the latency in the L3 model is introduced by the Nexus 7000 the Cisco design performs much better in this model, contributing only ~10 microseconds of latency to the forwarding path. Arista is still at 600 nanoseconds - 16x faster.

Figure-3:

Similar to Figure-2, in Figure-4 the Virtual Machine Forwarding Path diagram will require all frames to go to through the following systems Nexus 1000 --> Nexus 2000 --> Nexus 5000 --> Nexus 7000 --> Nexus 5000 --> Nexus 2000 --> Nexus 1000. The latencies and oversubscription rations will be consistent to slightly higher than in Figure-2 as well.

Arista continues to make the intelligent Layer-2 or Layer-3 forwarding decision, at the first switch port.

The Arista switches also provide Virtual Machine discovery mapping VM's to physical switch interfaces dynamically.

Figure-4:

Figure-5 shows the forwarding path for a vMotion live migration of a virtual machine, from one server to another, again in the same cabinet enclosure.

Moving a virtual machine involves sending a lot of data from one hot to another, and the faster it can be done, the smoother the vMotion operation is. Latency and bandwidth come into play in how smoothly a VM can be statefully moved.

Figure-5:

In the below video data sheet, Anshul Sadana - Arista vice president of customer and systems engineering, provides a closer look at the new Arista 7500 series.

Get the Flash Player to see this player.