QoS Support in High-Speed, Wormhole Routing Networks

Mario Gerla, B. Kannan, Bruce Kwan, Prasasth Palanti,Simon

Walton

Overview

• Introduction

• QoS via separate subnets

• QoS via synchronous framework

• QoS via virtual channels

• Conclusions

Introduction• Wormhole routing offers low latency, high speed

interconnection for supercomputers and clusters.• It’s a modification of virtual cut-through:

-A packet is forwarded to output port once its head is received at the switch

-If channel is busy, whole packet is buffered at input port

-Wormhole: packet composed of several flits is stored across several

switches

• Used in high speed LANs like Myrinet:-Asynchronous LAN

-uses wormhole routing, source routing, backpressure flow control to

achieve low latency and high bandwidth

Supercomputer SuperNet

• Two level architecture:– Optical backbone based on physical optical star topology

– High speed wormhole routing are Myrinet LANs

– Optical Channel Interface connects electronic LANs to optical backbone

• Provide support for distributed supercomputing.– Scientific visualization, video display, parallel applications

– Different types of traffic (low latency datagram, high bandwidth connection oriented)

– Different types of QoS

Objective

• Want to provide connection oriented traffic with QoS parameters:-reliable support: no worm loss

-scalable and deadlock free network

• Assumption:-Traffic with QoS is connection oriented

-QoS parameters specified at connection setup

-Connection can be refused if no guarantee for Qos parameters

-QoS parameters: average bandwidth, end-to-end delay or jitter

QoS support via Separate Subnet

• Create two subnets– One carries QoS traffic

– Another carries non QoS traffic

– Routing is independent for the subnets (Myrinet has support)

• Issues:– Call admission and control

– Source host behavior

– Number of interfaces at host

Call Admission & Control• Admission agent maintains state of QoS subnet

• Request for QoS traffic connection comes in

• Upon receiving request, agent decides a suitable route

• If route not available, host can retry or use other subnet

• If route exists, connection is accepted, host can send

• Once completed, host informs admission agent

• Admission agent update its view or state of subnet

Host Behavior

• Host must be responsible for amount of traffic injected in subnet according to QoS parameters it required

• Solution: host uses pacing mechanism– Allow only predetermined number of flits to be transmitted per

time period

Number of Interfaces

• Suppose host has only one interface

• Sender side:– Host can schedule transmission into the network

• Receiver side:– Possible non-QoS worm may block QoS worm– QoS worm encounters delay if non-QoS worm is large

• Solutions:– Two interfaces: this double cost of network– Account for the worst case non-QoS traffic delay on single host

interface at call setup time

Alternative• Subnets:

– difficult to provide delay bounds due to delay dynamics from blocking at different cross points

• Alternative:– Impose synchronous structure on top of the asynchronous network

– Enables control over the blocking

– Delay bounds and message priorities may be implemented

• Trade off:– Network is no longer asynchronous

– Under low traffic load, messages suffer delay due to synchronous protocol overhead

QoS support via Synchronous framework

• Similar to dedicated traffic channels

• Use timed-token to control traffic streams

• Provides tighter delay bounds and bandwidth guarantees

• Target Token Rotation Time TTRT limit the amount of transmission

• Average delay = TTRT

• Worst case delay = 2 * TTRT

How to support timed token

• Dedicated unidirectional ring is embedded in the network

• Attributes of Token scheme: -fair -deadlock free

Issues• Number of host interfaces

– Caused by interaction of QoS & non-QoS traffic– QoS traffic travel on core ring while non-QoS

travel on other links not on ring– If host with one interface is busy receiving non-

QoS message, a QoS message will suffer delay– QoS message must have preemptive priority

• To increase non-QoS throughput, embedded ring may be used if bandwidth is not completely taken by QoS traffic

Continued• Scalability:

– throughput performance maintained by increasing TTRT parameter

– Allows nodes to transmit for longer time when they have the token

– Causes less capability to provide tight delay bounds

Virtual Channel Based QoS

• Each link is split into two different sets of virtual channels used for datagram and QoS traffic

• Each input port buffer of switch is split into several disjoint buffers

• Link between node and input port of switch is a collection of virtual channels

• Allows worms to be interleaved

• Give QoS traffic priority in the network

Non-preemptive priority• A worm arriving at QoS virtual channel does not get

transmitted right away

• Current worm (datagram or QoS) being transmitted on outgoing link must either complete or get blocked

• Then, scan QoS virtual channels before datagram channels to schedule the worm for transmission on outgoing link

• Easy to apply preemptive priority by making arrived worm preempt datagram worm at the QoS virtual channel

Implementation• Preemptive and non-preemptive implementation

require intelligence switch

• At a switch:– monitor all traffic passing– Schedule QoS & non-QoS traffic according to protocol

• Harder to implement preemtive:– Switch must check arrival of QoS traffic at any input

port before transmitting non-QoS flit from output port

Advantage of virtual channels

• Network appears the same for both traffic

• Intelligent switches allocate bandwidth as required to support QoS

• Can provide delay jitter bounds

• Bandwidth guarantee is provided by employing call admission agent

conclusions• Wormhole routing networks provide low latency, high

bandwidth support for datagram traffic

• To support QoS traffic is a challenge

• Dedicated QoS subnet with pacing and call admission control can support QoS

• Synchronous framework on top of asynchronous network provides guaranteed bandwidth and delay

• Virtual channels with priority mechanism also is effective way to support QoS