Circuit switching in the Circuit switching in the InternetInternet

Ph.D. oral examinationPablo Molinero-Fernández

21st May 2002

2

Traffic doubles every yearTraffic doubles every yearRelative preformance increase

0

250

500

750

1000

2002 2004 2006 2008 2010 2012

Moore’s law

x2/ 1.5 years

Router capacity

x2.2/ 1.5 years

Traffi c growth

x2/ 1 year

1

Cost and complexity of five times as

many central offices is prohibitive

GAP OF 5x!!

3

Optics removes Optics removes bandwidth constraintsbandwidth constraints

But we cannot buffer light!

4

Circuit switchingCircuit switching

•Link bandwidth is reserved•Need signaling•No buffering•No processing in data path

5

Packet switchingPacket switching

•Link bandwidth is shared•Buffering•Per-packet processing

6

ContributionsContributions

The Internet needs circuit

switchingin the core

Provisioning of fat pipes in an all-optical

backbone

TCP Switching: how to

integrate circuit switching in the

core

7

How we think the How we think the Internet isInternet is

8

Why the Internet usesWhy the Internet usespacket switching packet switching

• Efficient use of expensive links:– “Circuit switching is rarely used for data

networks, ... because of very inefficient use of the links” – Bertsekas & Gallager ‘92

• Resilience to failure of links & routers:– ”For high reliability, ... [the Internet] was to

be a datagram subnet, so if some lines and [routers] were destroyed, messages could be ... rerouted” – Tanenbaum ‘96

9

What the Internet is What the Internet is really like todayreally like today

SONET/SDHDWDM

10

What the Internet is What the Internet is really like todayreally like today

Modems, DSL

11

Users

Performance criteriaPerformance criteria

Carriers•Cost: Bandwidth efficiency•Reliability and stability•Low complexity

•Response time•Service Level Agreement (QoS)

12

Users •Response time•Service Level Agreement (QoS)

What users wantWhat users want

Carriers•Cost: Bandwidth efficiency•Reliability and stability•Low complexity

13

Response time of Response time of packets and circuitspackets and circuits

Packet swCircuit sw 10 Mb/s1 Gb/sFlow

bandwidth1 s0.51 sAvg response

time1 s1 sMax response

time

All but one of circuits

finish earlier

File = 10Mbit

x 100

1 server100 clients

1 Gb/s

1410.99 sec10.99 sMax response time

Packet swCircuit sw 10Mb/

s;1Gb/s1 Gb/sFlow

bandwidth 1.10 sec10.50 sAvg response

time

A big flow can kill CS if it

blocks the link

File = 10Gbit/10Mbit

x 99

1 server100 clients

1 Gb/s

Response time when Response time when blocking occursblocking occurs

15

Response time with Response time with flow rate limitsflow rate limits

Packet swCircuit sw 1 Mb/s1 Mb/sFlow

bandwidth

10,000 sec 10,000 sMax response time

109.9sec 109.9sAvg response time

No difference between

CS and PS in core

x 99

File = 10Gbit/10Mbit

1 server100 clients

1 Gb/s

1 Mb/s

16

Analytical modelingAnalytical modeling

• Fluid model:

• Many independent arrivals => Poisson• Service policies:

– Packets: Processor Sharing– Circuits: FCFS

• Service time distribution:– Flow size variance: Bimodal– Realistic flow size distrib: Pareto xxf )(

pBPpAP 1)(;)(

17

Fluid model: M/BiModal/NFluid model: M/BiModal/N

•Access link is as fast as the core link

•Hogging by long flows

Flow size variance +

FCFS (load = 0.5)

0.1

1

10

100

0 0.5 1

Res

pons

e ti

me

rel.

to

Proc

esso

r S

har

ing

N=1

18

Fluid model: M/BiModal/NFluid model: M/BiModal/N

• Flow rate limited by access link (1/N)

• Same response time regardless of flow size variance, if N is large

Flow size variance +

FCFS (load = 0.5)

0.1

1

10

100

0 0.5 1

Res

pons

e ti

me

rel.

to

Proc

esso

r S

har

ing

1

32

N=2k

19

Fluid model: M/Pareto/NFluid model: M/Pareto/N

Link hogging becomes

very bad with heavy-tailed traffic, if ratio

N=1

FCFS (alpha = 1.3)

1

10

100

1000

10000

0 0.5 1load

Resp

onse

tim

e r

el.

to

Proc

ess

or S

har

ing

N=1

20

Fluid model: M/Pareto/NFluid model: M/Pareto/N

• Response time similar to IP if core/access ratio N is large

• Typically N >> 1,000

FCFS (alpha = 1.3)

1

10

100

1000

10000

0 0.5 1load

Resp

onse

tim

e r

el.

to

Proc

ess

or S

har

ing

1

2

4

816

256

N =2k

512

21

Users see little difference Users see little difference in response timein response time

•Simulation of full networks

•N is large => same response time1

10

100

1000

1E+2 1E+4 1E+6fl ow size (bytes)

resp

onse

tim

e (s

)

Circuit switching Packet

switching

22

Service Level Service Level AgreementsAgreements

• Packet switching:– Algorithms (WFQ, DRR, …) => not

used– Thus we must overprovisioning =>

used and it works

• Circuit switching:– Simple QoS: guaranteed BW => no

jitter

23

Users •Response time•Service Level Agreement (QoS)

What carriers wantWhat carriers want

Carriers•Cost: Bandwidth efficiency•Reliability and stability•Low complexity

24

Cost: Bandwidth Cost: Bandwidth efficiencyefficiency

• Argument: packets share all link BW => statistical multiplexing gain => more throughput with bursty traffic

• Reality: – Internet avg. link utilization: 5-20%

[Coffman&Odlyzko’02]

– Phone avg. link utilization: ~33% [Odlyzko’99]

– There is a glut of BW in the core [WSJ’00]

• Result:– Packets more efficient, but BW is no longer

a scarce resource

25

OC-12OC-3 OC-48

Carriers peak allocate Carriers peak allocate their networktheir network

• When over-provisioning, link BW is virtually peak allocated

• That is exactly what circuit switching does

Source: Chuck Fraleigh ‘02

26

Reliability and Reliability and stability (I)stability (I)

• Argument: because of the state, rerouting a circuit is more costly than with packets

• Reality:– Internet average availability:

1220 min/year down time [Labovitz’99]

– Phone average availability: 5 min/year down time [Kuhn’97]

27

Reliability and Reliability and stability (II)stability (II)

• Reality (cont.):– IP recovers in about 3 min (median),

sometimes it takes over 15 min [Labovitz’01]

– SONET required to recover in less than 50 ms

• Result: – No evidence packet switching is more

robust

28

Low complexity (I)Low complexity (I)

• Argument: No per-flow state => packet switching is simpler

• Reality: – PS: 8M lines of code in core router

[Cisco’s IOS ‘00]– CS: 18M lines of code in telephone switch

[AT&T/Lucent 5ESS ‘96]– CS: 3M lines of code in transport switch

[’01]

• Result: – Packet switching does not seem

inherently less complex than circuit switching

29

Functions in a Functions in a packet switchpacket switch

Interconnect scheduling

Route lookup

TTL proces

sing

Buffering

Buffering

QoS schedu

ling

Control plane

Ingress linecard Egress linecardInterconnect

Framing

Framing

Data path

Control path

Scheduling path

30

Functions in a Functions in a circuit switchcircuit switch

Interconnect scheduling

Control plane

Interconnect

Framing

Framing

Ingress linecar

d

Egress linecard

Data path

Control path

31

Low complexity (II)Low complexity (II)

• Argument: IP does not have the signaling of circuits switches => Routers go faster• Reality: – IP does almost same operations on every

packet as a circuit switch on the circuit establishment

– CS has no work to do once circuit is established

• Result: – The fastest commercially-available circuit

switches [Ciena ’01, Lucent ‘01] have 5x the capacity of the fastest routers [Cisco ’01, Juniper ’02]

32

Network architectureNetwork architecture

LANs &wireless

WANs

•Use packet switching •Better response time (ratio N small)•Efficient use of the spectrum

•Use circuit switching•More capacity, reliability•Similar response time & QoS

MANs•If metro-to-access BW ratio (N) is

small => use packets•Otherwise use what costs less

33

ContributionsContributions

The Internet needs circuit

switchingin the core

Provisioning of fat pipes in an all-optical

backbone

TCP Switching: how to

integrate circuit switching in the

core

34

Integration of circuits Integration of circuits and packetsand packets

• Create a separate circuit for each user flow

• IP controls circuits• Optimize for the most

common case– TCP (90-95% of traffic)– Data (9 out of 10 pkts)

TCP Switching

35

TCP Switching exposes TCP Switching exposes circuits to IPcircuits to IP

TCP Switches

IP routers

36

TCP “creates” a TCP “creates” a connectionconnection

Router Router RouterDestina

-tionSource

SYN

SYN+ACK

DATA

Packets Packets

PacketsPackets

37

Let TCP leave state Let TCP leave state behindbehind

Boundary TCP-SW

Core TCP-SW

Boundary TCP-

SW

Destina-tionSource

SYN

SYN+ACK

DATA

Create circuit

One Circuit PacketsPackets

Create circuit

38

What is a typical flow?What is a typical flow?

• Most traffic are TCP connections:– Taking less than 10 s, 12 packets and

4 KBytes– Obtaining less than 100 Kbps– ~40% of the flows continue

transmitting ACKs after sending a FIN (asymmetrical closures)

39

State management State management feasibilityfeasibility

• Amount of state– Minimum circuit = 56 Kb/s.– 178,000 circuits for OC-192.

• Update rate– About 51,000 entries per sec for

OC-192• Implementable in hardware or

software.

40

TCP Switching can be TCP Switching can be implemented in implemented in

softwaresoftwareTCP Switching boundary router:• Kernel module in Linux 2.4 1GHz

PC • Forwarding latency

– Forward one packet: 21s.– Compare to: 17s for IP. – Compare to: 95s for IP + QoS.

• Time to create new circuit: 57s. Source: Byung-Gon Chun ‘01

41

Slow Start

Congestion Avoidance

Flow duration

Inactivity timeout

Cir

cuit

BW

Inst

an

tan

eou

s b

an

dw

idth

time

Bandwidth inefficienciesBandwidth inefficiencies

Compromise: inactivity timeout of

few seconds

42

Related workRelated work

• IP Switching– Uses ATM virtual circuits (i.e. packets)– Became MPLS (but no longer user

flows)

• Generalized Multi-Protocol Label Switching (GMPLS)– Coarse circuits– Heavy weight signaling

43

ContributionsContributions

The Internet needs circuit

switchingin the core

TCP Switching: how to

integrate circuit switching in the

core

Provisioning of fat pipes in an all-optical

backbone

44

New networking New networking scenarioscenario

Optical Switches

IP routers

45

New networking New networking scenarioscenario

Optical Crossconnect

IP Linecards

46

Circuit creation is slowCircuit creation is slow

• We need a safeguard to avoid running out of BW => inefficiency

A slow signaling requires a larger BW safeguard

47

Controlling coarse Controlling coarse circuits with user flowscircuits with user flows

• Should use the fastest optical switching elements

• Should avoid ACKs => no RTT

0.0001

0.001

0.01

0.1

1

1E+04 1E+06 1E+08Bandwidth saf eguard (bps)

Prob

. of

exce

edin

g

safe

guar

d

10 us100us

1 ms

10 ms

100 ms

1 s

Circuit

creation

latency

48

ConclusionsConclusions

• Circuits should be used in the core, packets in the edges

• TCP Switching integrates circuits and packets in an evolutionary way

• User flows can be used to control an all-optical network

49

PapersPapers

• PMF, Nick McKeown, "TCP Switching: Exposing Circuits to IP“, IEEE Micro, 2002

• PMF, NM, "TCP Switching: Exposing Circuits to IP“, Hot Interconnects, 2001

• PMF, NM, “Study of routing behavior trough traffic analysis and traceroute measurements”, NAT Times, 2001

• PMF, NM, Hui Zhang, "Is IP going to take over the world (of communications)?“, submitted

Thank youThank you