8/7/2019 Renesys. Egypt Leaves The Internet

http://slidepdf.com/reader/full/renesys-egypt-leaves-the-internet 1/18

Egypt Leaves The Internet

Jim Cowie

NANOG51, Miami

1 February 2011

8/7/2019 Renesys. Egypt Leaves The Internet

http://slidepdf.com/reader/full/renesys-egypt-leaves-the-internet 2/18

Credit: Arbor Netwiorks

0

2

4

6

8

10

12

14

16

Gb

os

Week of January 17, 2011

HTTP HTTPS Flash HTTP-Alt SMTP Yahoo Messenger

What Egypt Traffic Normally Looks Like

Credit: Arbor Netwiorks

10-15 Gbps and almost entirely inbound traffic (95% +)

Through four upstreams

Looks like normal consumer network in US / Europe

Traffic Out of Egypt

8/7/2019 Renesys. Egypt Leaves The Internet

http://slidepdf.com/reader/full/renesys-egypt-leaves-the-internet 3/18

Traffic In to Egypt

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

Gbos

Week of January 17, 2011

HTTP HTTPS Flash HTTP-Alt SMTP Yahoo Messenger

What Egypt Traffic Normally Looks Like

Credit: Arbor Netwiorks

8/7/2019 Renesys. Egypt Leaves The Internet

http://slidepdf.com/reader/full/renesys-egypt-leaves-the-internet 4/18

Egypt Traffic Disruption (Day View)

Normally estimate 3-5 Gbps of traffic

Dropped to several hundred Mbps January 27

Further dropped to tens of Mbps last night

8/7/2019 Renesys. Egypt Leaves The Internet

http://slidepdf.com/reader/full/renesys-egypt-leaves-the-internet 5/18

Underlying Problem: What Happened?

Shortly after midnight Friday in Cairo, Renesys

observed EG providers withdrawing BGP routes

from the global table

Egypt went from appx 3500 routed prefixes to

about 150 routed prefixes in the space of 20min

Global BGP convergence on the order of 30-60

seconds Interprovider spacings of first onset were longer:

a minute to several minutes apart

© 2011 Renesys Corporation NANOG51 1 Feb 2011 5

8/7/2019 Renesys. Egypt Leaves The Internet

http://slidepdf.com/reader/full/renesys-egypt-leaves-the-internet 6/18

Mirjam Kühne, NANOG 51, 1 Feb. 2011

http://stat.ripe.net/egypt

6

8/7/2019 Renesys. Egypt Leaves The Internet

http://slidepdf.com/reader/full/renesys-egypt-leaves-the-internet 7/18

© 2011 Renesys Corporation NANOG51 1 Feb 2011 7

Grey: routed

Black: unrouted

Orange: Egypt

8/7/2019 Renesys. Egypt Leaves The Internet

http://slidepdf.com/reader/full/renesys-egypt-leaves-the-internet 8/18

Before and After: A Hole In The Internet

© 2011 Renesys Corporation NANOG51 1 Feb 2011 8

8/7/2019 Renesys. Egypt Leaves The Internet

http://slidepdf.com/reader/full/renesys-egypt-leaves-the-internet 9/18

Staggered downtimes: no kill switch?

© 2011 Renesys Corporation NANOG51 1 Feb 2011 9

8/7/2019 Renesys. Egypt Leaves The Internet

http://slidepdf.com/reader/full/renesys-egypt-leaves-the-internet 10/18

No clear signs of congestion beforehand

© 2011 Renesys Corporation NANOG51 1 Feb 2011 10

8/7/2019 Renesys. Egypt Leaves The Internet

http://slidepdf.com/reader/full/renesys-egypt-leaves-the-internet 11/18

Same view from the inside

© 2011 Renesys Corporation NANOG51 1 Feb 2011 11

Note 180ms

medianlatencies for

domestic

routes underNORMAL

conditions

8/7/2019 Renesys. Egypt Leaves The Internet

http://slidepdf.com/reader/full/renesys-egypt-leaves-the-internet 12/18

Who was left?

© 2011 Renesys Corporation NANOG51 1 Feb 2011 12

Prefixes remained alive from several sources

Traffic continued to flow at minimal levels

Inbound traces to routed blocks now die on

Seabone at Palermo, suggesting additionalproblems; routing not sufficient for connectivity

Lots of sites have DNS issues and other 

reachability issues, even if routes remain

Eg., stock exchange up, settlement down

Some IPv6 remained (nobody noticed)

8/7/2019 Renesys. Egypt Leaves The Internet

http://slidepdf.com/reader/full/renesys-egypt-leaves-the-internet 13/18

Monday, 20:27 UTC: Noor Group down

© 2011 Renesys Corporation NANOG51 1 Feb 2011 13

8/7/2019 Renesys. Egypt Leaves The Internet

http://slidepdf.com/reader/full/renesys-egypt-leaves-the-internet 14/18

Monday, 20:27 UTC: Noor Group down

© 2011 Renesys Corporation NANOG51 1 Feb 2011 14

Noor customerssupport EGfinancial industry,market operations

8/7/2019 Renesys. Egypt Leaves The Internet

http://slidepdf.com/reader/full/renesys-egypt-leaves-the-internet 15/18

Tuesday Status

© 2011 Renesys Corporation NANOG51 1 Feb 2011 15

About 75 prefixes still up, but most traceroutes

die at Palermo on Seabone

online and reachable

This is very different from an Egyptian cable cut

No signs of problems on SMW3, SMW4, FLAG,

etc. for regional providers downstream

Renesys continuing to monitor regional networks

carefully for signs of instability or outage

8/7/2019 Renesys. Egypt Leaves The Internet

http://slidepdf.com/reader/full/renesys-egypt-leaves-the-internet 16/18

Visualization: Egypt Outage

© 2011 Renesys Corporation NANOG51 1 Feb 2011 16

http://www.youtube.com/watch?v=b_jRcxuemtg

60x playback; one second per minute

Announced routes depicted per origin ASN

Withdrawal events flash white

Piecewise Hilbert curve map depicts actual

Individual providers are color-coded

Fades to black as Egypt leaves the Internet

8/7/2019 Renesys. Egypt Leaves The Internet

http://slidepdf.com/reader/full/renesys-egypt-leaves-the-internet 17/18

Video: Visualization of Original Outage

© 2011 Renesys Corporation NANOG51 1 Feb 2011 17

8/7/2019 Renesys. Egypt Leaves The Internet

http://slidepdf.com/reader/full/renesys-egypt-leaves-the-internet 18/18

Thank You!

www.renesys.com