Fat-trees and Dragonflies – A perspective

on topologies

Sven-Arne Reinemo

svenar@simula.no

Simula Research Laboratory

HPC Advisory Council Switzerland Workshop

March 15, 2012

o Frank Olaf Sem-Jacobsen

o Mathias Myrland

o Professor Tor Skeie

o Eitan Zahavi, Mellanox

o Gilad Shainer, Mellanox

o Motivation

o The fat-tree topology

o The dragonfly topology

o Group topologies for the dragonfly

o Blocking, cost, and scalability

o Conclusion

o The fat-tree is the dominating topology for InfiniBand networks, but the recently proposed dragonfly topology has been suggested as an alternative.

o In that context we would like to answer the following questions:

o Is the dragonfly a viable alternative to the fat-tree?

o How do they compare in fundamental properties such as blocking and scalability?

o What about cost?

o What dragonfly configurations are attractive?

oMost common topology for InfiniBand clusters.

oCan be routed deadlock free without additional

resources such as virtual lanes.

oFault-tolerant through its path diversity.

oFull bisection bandwidth for arbitrary

permutations.

oScalable, but costly when going beyond to

levels.

oPerformance suffer slightly due to static

routing.

m-port n-tree k-ary n-tree hybrid tree

o Recently proposed by John Kim et al. in [1].

o Caused discussion in the HPC community about is suitability for IB and as an exascale topology.

o The dragonfly is a hierarchical topology with the following properties:

o Several groups are connected together using all to all links, i.e. each group has at least one link directly to each other group.

o The topology inside each group can be any topology. The recommendation in [1] is the flattened butterfly.

o Focus on reducing the number of long links and network diameter.

o Requires non-minimal global adaptive routing and advanced congestion look ahead for efficient operation.

[1] John Kim et al. “Technology-Driven, Highly-Scalable Dragonfly Topology” in proceedings of the

35th

International Symposium on Computer Architecture, 2008.

The recommendation in [1] is to keep:

Inter group

topology

Group0

Groupg-1

Group1

Switch

0

Switch

1

Switcha-1

0 …. p-1

0 …. h-1

a * h links to other groups

Intra group

topology

a ³ 2p ³ 2h

Dragonfly conforming to versus an unrestricted

dragonfly. a ³ 2p ³ 2h

In our studies we will:

oconsider the flattened butterfly as a group

topology.

oconsider the 2d torus as a group

topology.

oconsider a fat-tree as a group topology.

orelax the requirement to

compare it on equal terms with the fat-

tree.

a ³ 2p ³ 2h

A single flattened butterfly group with 8 terminals and 8 external

connections.

Fully connected, but requires non-minimal routing for path

diversity and load balancing.

S0 S1 S2 S3

Statistics for 100 random permutations on a 25 760 ports

dragonfly with flattened butterfly groups, buildt using 36-port

switches.

Good for nearest neighbour traffic, but bisection bandwidth does

not scale for arbitrary traffic.

Statistics for 100 random permutations on a 25 760 ports

dragonfly with 2d torus groups, built using 36-port switches.

Rather than building a complete fat-tree we use the fat-tree as

the group topology in the dragonfly.

oAs shown, the viable dragonfly groups are

the a flattened butterfly and the fat-tree.

oWe will compare these dragonfly

configurations with a 3- and 4-level fat-tree.

oWe look at how the blocking and cost of

these topologies develop as the size

increases.

Comparing the dragonflies with a 3- and 4-tier fat-tree.

Scaling of the 4-tier fat-tree and two hybrid dragonflies.

Key results:

o The topology with best cost/performance depends on the size.

o The fat-tree gives the best cost/worst-case-blocking

performance.

o The dragonfly is almost as good as the fat-tree when

considering cost/average-blocking performance.

Remember:

o The dragonfly require support for adaptive routing and

congestion look ahead for optimal behaviour, this is not

supported by any existing hardware.

o The dragonfly requires multiple virtual channels for deadlock

avoidance.