Benchmarking Broadband Internet with Bismark

Srikanth Sundaresan, Nick FeamsterGeorgia Tech, Atlanta, USA

Walter de Donato, Antonio PescapéUniversity of Napoli Federico II, Napoli, Italy

Renata TeixeiraUniversity Pierre et Marie Curie, Paris, France

Broadband Connectivity is exploding

● 200 million Americans have broadband

● National Broadband plan aims to increase that

● But, what is current experience with broadband?

Talk Outline

● Questions about broadband performance● Bismark: our measurement platform● Some initial results● Future work

Are users really getting what they pay for?

Study in France confirms this view

● Over 20,000 users across major ISPs, geographical regions

● Latency and throughput study from end hosts

6

Significant performance variation

Cum

ulative

fraction o

f use

rs

Fewer than half of the users achieve 80% of

advertised SLA. Why?

S. Sundaresan, L. Di Cioccio, N. Feamster, R. Teixeira. “Which Factors Affect Home Network Performance?”

● ISP, service plan, geographical location have effect.

● Gap between plan rate and achieved rates

How do we study access link performance?

● Study from outside– Dischinger et al (IMC 2008), Netalyzr (IMC

2010)

– Not continuous, not many per user, no view into home

● Study from inside– Grenouille project

– Hard to account for device diversity

– Hard to account for home network

We don't really understand home networks

● How much of user performance is affected by factors outside ISP's control?

● Home networks have a big impact– Network setup – multiple diverse devices

– WiFi effect

– Competing traffic

– User behavior

Talk Outline

● Questions about broadband performance

● Bismark: our measurement platform

● Some initial results● Future work

Bismark

● Deploy programmable gateways in homes● Noxbox deployment: about 35 around Atlanta● SamKnows deployment: about 11000 around

the US

NoxBox Netgear

The design

Internet

MeasurementServer

MeasurementServer

ManagementServerDB

Architecture Overview

Router

Router

Vantage point between the home and the ISP

● Observes all traffic passing through network● Isolate individual factors affecting network

performance– Effect of home network

– Wireless

– Access link

Measurements - Active

● Latency, Jitter and Packet Loss– Last mile and end-to-end latency (ping)

– Upstream and downstream jitter, packet loss (DITG)

– Important for gaming, VoIP

● Throughput– Upstream and downstream (curl, shaperprobe)

– Capacity, available bandwidth of access link

● DNS lookup time and faults– Affects user performance

Measurements – Passive

● ARP and DHCP logs– Nature of home network

● Aggregate flow statistics– Incoming and outgoing (TIE)

– Understand usage patterns, available bandwidth

● Wireless stations– Access point configurations (airodump-ng)

– Understand home wireless networks

Current Deployment

● 10 boxes deployed● 7 in ATT, 2 in Comcast, 1 ClearWire● Most of the deployments within Atlanta● All measurements done to server in GaTech

Talk Outline

● Questions about broadband performance● Bismark: our measurement platform

● Some initial results● Future work

Overview of results

● Initial look at latency from gateway to:– First IP hop in ISP network (last mile)

– Server in GaTech

– High variation across homes, ISPs

● Initial throughput analysis– Single threaded HTTP throughput using curl

– Path capacity using shaperprobe

Latencies vary within same ISPR

TT

(ms )

RT

T(m

s )

Baselines Different for 2 ATT customers

Access link technology has impact

● High variation in WiMax and Cable● ADSL latencies are more tightly bound

RT

T(m

s )

RT

T(m

s )

Comcast Clear

Last mile latencies dominate● All but 2ms comes from last mile● ADSL last mile – 8 to 25ms, WiMAX – ~ 75ms!

Very high correlation (0.95) to end-to-end latency

End-to-end latency

Last mile latency

Plan rate

Kbps

CD

F

Single ATT customer

Single threaded TCP downloads consistently see about 80% of promised capacity

Dowstream throughput doesn't quite reach maximum limit

Why not?

● Single threaded throughput error?● Multiple parallel TCP sessions see higher

throughput (> 95% for ATT)– However, highly disruptive to user

● Capacity measurements also close to limit● We are interested in realistic usage scenarios

– What does a normal user with normal usage patterns see?

Throughput Consistently greater for ATT

Plan rate

Single ATT customerKbps

CD

F

Upstream throughput exceeds advertised limits

Key findings

● Latency is important– Baseline varies

– Highly dependent on access link technology

– Last mile latency dominates, highly correlated to end-to-end latency

● Downloads are consistently lower than promised rate

● Uploads are consistently higher

Talk Outline

● Questions about broadband performance● Bismark: our measurement platform● Some initial results

● Future work

Future Deployment and Measurements

● Larger deployment for critical amount of data● Variety in ISP, service plans, geographical

locations– Working with SamKnows

● User study to understand– What users expect from their Internet

– How home networks are set up

● How can we improve user experience?

More questions about performance

● Does application affect performance?● How do different factors affect performance?

– Latency, ISP, service plan

● Better statistical tools to analyze current data● What would you like to know/measure?

Outsourcing Home Network Configuration and Management

● What prevents untrusted tools from:– Sniffing my traffic?

– Consuming all my bandwidth?

– Using my printer?

● Solution: Hold tools accountable to a management policy

● PlanetLab-like testbed for home networks?

Thanks!

srikanth@gatech.edu