Myricom’s newest Myri-10G switches are based on a 110ns-latency 32-port Myrinet-protocol crossbar-switch chip. The photographs above show the current choice of enclosures for these modular switches. Different mixes of Ethernet or Myrinet external ports, as well as ports with different PHYs, are configured by using different types of line cards.

Before getting into the details of configuring these modular switches, let us offer a top-level view. The line cards plug into a backplane that connects the crossbar-switch chips on the line cards in a diameter-3 full-bisection Clos network. The characteristics of the Clos topology together with the adaptive-routing techniques implemented in Myricom software and firmware result in a switch network that closely approaches the ideal in delivered throughput independent of the traffic patterns.

For the 10G-21U-CLOS-ENCL for up to 512 host ports, the pre-wired Clos network in the backplane provides the topology below, in which the circles represent 32-port crossbar-switch (XBar32) chips on the line cards. The 16 XBar32s across the middle row are referred to as the "spine" of the Clos network. For the spine two XBar32 chips are packaged on each of eight 10G-2SW32LC line cards, which have no front-panel ports. You can see in the photograph above of the 21U enclosure that these line cards for the spine of the Clos network are in the middle. The other 32 switches are packaged on 32 port line cards, which each have 16 ports through the front panel and 16 ports to the backplane.

256 Host Ports

256 Host Ports

The topology of the Clos network for the smaller switches is simply scaled down from the diagram above, with either one half or one quarter the number of XBar32 chips.

Let us now get to the product-code details of configuring these switches, which starts with a choice of three switch enclosures:

The enclosure provides the mechanical housing, power distribution, cooling, an internal monitoring network, and the pre-wired Clos network. A photograph of the 10G-21U-CLOS-ENCL for up to 512 host ports is shown below without line cards.

The top-most line-card slot, labeled M, is reserved for the 10G-MONITOR-B monitoring line card, which is included with the enclosure. Its dual 10/100/1000Base-T Ethernet ports provide access via a web browser and other software interfaces to the status of the switch: port states, packet traffic, errors, voltages, currents, temperatures, fan rotation rates, and more.

The 10G-POWER modules operate in an auto-parallel mode for load sharing and redundancy, and convert from 100-240 VAC input power to the 12 VDC power distributed in the enclosure. Each 10G-POWER module can supply up to 840W (12V @ 70A). When you select the line cards to be used in an enclosure, we recommend for redundancy and derating of the power modules that the total power is less than the power that can be supplied if one power module fails or is removed (N+1 redundancy).

The 10G-FAN-B units are on the front (air-inlet) side. The ports and power supplies are on the rear (exhaust air) side. You can see the fans in the photograph to the right through the split between the backplanes, which is part of the airflow path. Each 10G-FAN-B actually includes two fans in series for efficiency and redundancy, and includes monitoring and speed-control circuits. As you can see from the table above, the number of fan units is:

• 1 10G-FAN-B for the 10G-7U enclosures
• 2 10G-FAN-B for the 10G-12U enclosures
• 4 10G-FAN-B for the 10G-21U enclosures

The switch enclosure will normally be mounted in a rack using the 10G-RACK-KIT. The rack-mount kit is a pair of telescoping shelf brackets to support the enclosure in standard 19-inch racks.

As noted above, the center line-card slots are reserved in these Clos enclosures for 10G-2SW32LC line cards that provide the switches for the spine of the Clos network. You can see in the photograph the wide connectors for these line-card slots. Although you need to order these 10G-2SW32LC line cards separately, the Clos switches are generally shipped with these line cards already installed to increase the rigidity of the enclosure in shipping.

The remaining line-card slots, up to 8 for the 10G-7U-CLOS-ENCL, up to 16 for the 10G-12U-CLOS-ENCL, or up to 32 for the 10G-21U-CLOS-ENCL, are used for port line cards. The line-card slots below the monitoring card are numbered top-to-bottom 0-9 in the 10G-7U-CLOS-ENCL, 0-19 in the 10G-12U-CLOS-ENCL, and 0-39 in the 10G-21U-CLOS-ENCL. Because of the symmetries of the Clos network, any port line card can equally be used in any port-line-card slot. The alignment pins in the backplane and at the back of the line cards are keyed so that only a monitoring line card can be inserted in the M slot, only a 10G-2SW32LC line card in the center slots, and only port line cards in the remaining slots.

Although you will find switch line cards with 10G-THRU-* product codes in the product list, the only line cards that may be used in the port-line-card slots of Clos enclosures have XBar32 switch chips and product codes 10G-SW32LC-*.

Here is a list of the common and current types of line cards used in the Clos enclosures. Please refer to this Guide to Myri-10G PHYs (pdf) for a description of the types of ports.

For convenience in checking that the total power of your configuration is within the limits of the enclosure, we include above the maximum power used by each type of line card.

Power Calculations: Each 10G-POWER power module is rated for 840W (12V @ 70A). In order for an enclosure to have N+1 redundant power (tolerate the failure of one power module), the total power consumed should not exceed:

This upper limit on the power also assures that the power modules are de-rated for reliability. Note that the 12U and 7U enclosures may optionally be equipped with additional 10G-POWER power modules.

In order to check whether a configuration is within its power limit, you need to add up the power used by the fans and line cards. Each 10G-FAN-B unit uses 40W of 12VDC power.

If fewer than the full complement of transceivers is used on a line card, you can deduct the power allowed for them. After allowance for regulator efficiency (85-90%) on the line card, each QSFP transceiver consumes 1.1W, and each SFP+ transceiver consumes 1.1W. For example, a 10G-SW32LC-16QP with 6 transceivers, 10 omitted, has an actual power of 31W - 11W = 20W.

Example: 7U enclosure with 96 Ethernet-protocol SFP+ ports and 32 Myrinet-protocol QP ports:

The total power is well within the 840W limit for the 7U enclosure with two 10G-POWER modules, so we won't bother to deduct the power allowed for ports without transceivers.

Once the total 12V power consumed is calculated, divide by 0.8 (conservative) for the power-module efficiency to get the AC input power, e.g., 765W for the example above. You can then multiply the AC input power by 3.4 to get the BTUs/hour, e.g., 2601 BTU/hour for this example.

How to safely hot-swap everything: For safety, control of electromagnetic interference (EMI), and for airflow, any otherwise unused line-card slot must be filled with a 10G-AIRDAM line card. The monitoring line card and other line cards may be "hot swapped," i.e., replaced without turning the power off. However, line cards should be hot-swapped only one at a time. If two or more adjacent line cards are removed, there is an "energy hazard" due to the possibility of being able to reach the connectors carrying the full output of the multiple power modules. The redundant power modules (10G-POWER) are also individually hot-swappable. The 10G-FAN-B fan units are hot-swappable, but you should not reach inside the front of the switch enclosure when you remove a fan unit. When in doubt, remove the power, which with redundant power modules requires removing the power cords from all of the power modules.

For applications requiring more than 512 host ports, the switch networks are scalable by interconnecting the internal Myrinet fabrics. A full-bisection Clos network of 32-port crossbar switches for more than 512 host ports up to 8192 host ports has a diameter of 5 switches. The standard and recommended way to build such networks is to use diameter-3 Clos switches as the spine of the network, and 32-port edge switches.

Inasmuch as a large number of edge switches will be needed for large configurations, Myricom provides a way of packaging these switches at high density using 10G-*-EDGE-ENCL enclosures. These enclosures are the same as the 10G-*-CLOS-ENCL enclosures, but with a different backplane. The 10G-21U-EDGE-ENCL enclosure packages up to 20 32-port switches with common power, cooling , and monitoring. Each pair of line-card slots, 0, 1; 2, 3; ... ; 38, 39, is connected "back-to-back" for a pair of 10G-SW32LC-* and 10G-THRU-* line cards to form a 32-port switch. A typical pair may be a 10G-SW32LC-16ES for 16 Ethernet-protocol 10GBase-SR host ports and a 10G-THRU-16QP for 16 Myrinet-protocol connections to the Clos spine switches. Similarly, the 10G-12U-EDGE-ENCL packages up to 10 32-port switches, and the 10G-7U-EDGE-ENCL packages up to 5 32-port switches.

The same scheme can be applied to blade systems with a 32-port edge switch integrated into each blade chassis.

Due to the variety of special optimizations possible, please consult help@myri.com or sales@myri.com about configurations in excess of 512 host ports.