Researching 10Gbps switches


Jonathan Frappier Virtxpert

Recently I’ve had to research 10Gbps switches, not being an everyday network person I’d typically just ask Cisco and be done with it, after all as the saying goes no one ever gets fired for buying Cisco but wanted to look into what other vendors have for options.  The purpose of the blog post is not to suggest this is the end all be all, but provide some basic research to others and to open conversation as to anything I might be missing, or considerations I need to look at that I’ve not (again I’m not a full time network person).

This particular use case is for a small data center to support new servers using 10Gbps Ethernet for shared storage and networking.

Assumptions:

  • Each physical server will have no more than two 10Gbps interfaces
  • Future growth will likely not exceed 20 total physical servers for a total of 40 required ports.
  • The 10Gbps switch will handle L2 10Gbps traffic
  • Outbound VM network traffic will be routed to an existing core switch and firewall
  • Provide enough ports for future physical host considerations
  • Existing equipment with only 1Gbps links will stay connected to existing switches

To support redundancy I would like to have at least two physical switches to connect one each of the two 10Gbps server ports.  The expected physical layout would be similar to the image below:

10Gbps with existing network

Raw technical specifications were just about equal across the board, most of the switches to support the size of the deployment needed all had roughly the same throughput, PPS and MAC table support.  Cost was also fairly equal (note cost was pulled from public reseller sites, you should check with your reseller or VAR on pricing).  Here is what I looked at (crowd sourced list via Twitter; thank you Michael Davis, Rob Nelson, Ed Swindelles, Robert Novak):

  1.  Number of ports (and type)
  2. Available port speed
  3. Number of uplinks (and type)
  4. Available uplink speed
  5. Throughput
  6. Packets per second
  7. Latency
  8. CPU/Memory
  9. Buffers
  10. Port licensing
  11. Cable compatibility
  12. Warranty
  13. First year cost
  14. SFP cost
  15. Maintenance cost

A few things to note that aren’t listed here; all switches support LAG groups, the number of groups and group members vary generally between 16 and 32 ports per LAG group.  The SuperMicro switch supported 8 per LAG.

Thanks to @bvdlatisit for suggesting the 3524 (and thus 3548 – I’ll check those out and update this post shortly)

ManufacturerModelSizeRedundant PS# of PortsPort TypePort SpeedUplinksUplink TypeUplink SpeedThroughputPPSLatencyBuffersMAC TableWarrantySwitch CostTwinax 5M CostProduct Link
Cisco3064X1UYes48SFP+Up to 10Gbps4QSFP+4×10+40Gbps1.28Tbps950 mpps?9 MB shared1280001 Year18500280http://goo.gl/7dUrQG
Cisco3064T1UYes48RJ-45Up to 10Gbps4QSFP+4×10+40Gbps1.28Tbps950 mpps1.41 us9 MB shared1280001 Year18000280http://goo.gl/7dUrQG
Arista7124SX1UYes24SFP+Up to 10Gbps0480Gbps360 mpps500 ns??1 Year16500135http://goo.gl/B34DE7
Arista7050S-641UYes52SFP+Up to 10Gbps4QSFP+4×10+40Gbps1.28Tbps960 mpps1.2 us9 MB shared1280001 Year26000135http://goo.gl/Pu9U8C
JuniperQFX3500-48SQ-ACRB1UYes48SFP+ / SFPUp to 10Gbps4QSFP+4×10+40Gbps1.28Tbps960 mpps< 1 us9 MB shared1200001 Year25000170http://goo.gl/pqH4Bc
ExtremeX670V-48x – VIM4-40G4X1UYes48SFP+Up to 10Gbps4QSFP+4×10+40Gbps1.28Tbps952 mpps< 1 us9 MB shared1280001 Year21000?http://goo.gl/0oSlEn
ExtremeX670V-48x1UYes48SFP+Up to 10Gbps0960Gbps714 mpps< 1 us9 MB shared1280001 Year18000?http://goo.gl/0oSlEn
SupermicroX3348SR1UYes48SFP+Up to 10Gbps4QSFP+4×10+40Gbps1.28Tbps????1 Year13365156http://goo.gl/bbme0B
DellN40641UYes48SFP+Up to 10Gbps4QSFP+4×10+40Gbps1.28Tbps952 mpps?9 MB shared?Lifetime16825156http://goo.gl/XcEFuw
HP5900AF1UYes52SFP+Up to 10Gbps4QSFP+4×10+40Gbps1.28Tbps952 mpps< 1.5 us9 MB shared1280001 Year24200?http://goo.gl/ptXgWQ

Summary

At this point, I seem to have circled back to Cisco, specifically the 3064X which was also suggested by a re-seller I spoke to about pricing.  While the SuperMicro was the least expensive, not being a network person I’m not comfortable taking on the support of it directly (if some vendor slapped their logo on it and supported it directly that’s fine) and while Arista seems to be the new hotness, I also don’t have an unlimited budget so comparing features and price, Cisco seems to be it.  If you’ve deployed 10Gbps switches, what do you think of this information?  What did I miss? And what have your experiences been?

Definitions:

SFP+:   Enhanced Small Form-Factor Pluggable

QSFP+:  Quad SFP

ns:   nanosecond (http://goo.gl/M9LxEM)

us:  microsecond (http://goo.gl/M9LxEM)

Researching 10Gbps switches