Better Than Best Effort

Aldrin Isaac / CTO, Network Architecture @ Bloomberg

Reliability on the Internet

About Bloomberg

●  Market data and news ●  Serve global financial community ●  Revenue primarily from Bloomberg

Professional desktop product, followed by order management and trading products.

●  Large global private network. Most customers use this.

https://www.youtube.com/watch?v=LE8HiHZcgEE

2

Global private network o  3 Primary DCs o  16 Hub sites o  100+ Edge sites o  90+ Countries (customer locations)

o  22,000+ Leased lines Edge sites and regional hubs serve as network access points and also host caching and replication servers. Twin edge sites per city. Most customers connect to different local edge sites from paired managed CE over leased lines.

SP

LA

FF GE

RO

BE SY

TO

Customers

Primary DCs

3

Private MPLS-based WAN

Local VPNs Regional VPNs

Global VPNs

Private WAN MPLS-based NaaS

4

Compute

Data

Compute

Data

Local edge sites Regional hub sites Global DC sites

Data

Compute

BA

Private customer sites

PE Leased lines

VRF

5

Our customers need non-stop service

A sizable portion of mission-critical business connectivity happens over private lines. Delivering a highly available service over a private network is mostly achievable.

With the growing use of Internet-based SaaS and IaaS by our customers, Bloomberg needs to contend with the Internet as the access medium of choice by our customers.

Furthermore, with Internet access costs falling faster than leased line costs, requiring customers to purchase leased lines increases the cost of our service.

The challenge for SaaS providers such as Bloomberg is to find ways to deliver mission critical services over the Internet. We need to keep services operating non-stop in the face of problems that develop in infrastrastructure.

Changing dynamics -- Private => Internet

6

Internet Yin and Yang Yang -- Any service. Any user. Any content. Anywhere. Anytime. “Cheap” Yin -- Unreliable.

7

DDoS, BGP route leaks, DNS hijack, routing instability, MTU misconfigurations, flakey links, crippling congestion...

Incompatible for mission critical?

8

Achievable network visibility -- Internet vs Private

Like feeling vs seeing

9

How are the customers doing?

Vocal 10% Other 90% Cartoon by Sarah Cooper

TheCooperReview.com

10

So where do we start?

Fixing the Internet at a routing level isn't currently an option, at least not in the near future. Too many operators with mostly uncoordinated goals and interests and varying competency. Instead need to find and to promote ways of using the Internet that will significantly improve user experience.

11

Internet - the bad and good news

The bad news: The active path between any particular service point and a customer may be unpredictably impaired. The good news: Other paths are most likely available between that customer and [potential] service points that are good. HOW CAN WE USE THEM?

12

SP

LA

FF GE

RO

BE SY

TO

Customers

Primary DCs

SP

LA

FF GE

RO

BE SY

TO

Customers

Primary DCs

Internet

Internet access network

Our first step - Increase points of presence Establish local presence through integration with private WAN

13

WAN with Integrated Internet

Local VPNs Regional VPNs

Global VPNs

IBRs

Internet

BLBs

14

Private WAN MPLS-based NaaS

Compute

Data

Compute

Data

Local edge sites Regional hub sites Global DC sites

Data

Compute

BA

Private customer sites

PE Leased lines

VRF

Low hanging fruit

Anycast ●  Proximity measured in AS hops. Shot in the dark. ●  If closest anycast address doesn’t work for an endpoint, tough luck. ●  Good for DNS. Not really good for stateful application sessions.

FEC ●  Good for somewhat lossy paths up until loss threshold ●  Requires tunnelling TCP in datagrams. AFAIK no native TCP FEC ●  Westwood+ congestion control algorithm may achieve comparable performance

https://wiki.ittc.ku.edu/resilinets_wiki/images/Baldantoni-Lundqvist-Karlsson-2004.pdf

●  Bandwidth/latency tradeoff

15

Low hanging fruit

CDN / Akamai SureRoute IP ●  Works pretty good ●  Pay-per-byte ●  Still need intelligence in endpoints to support distributed origins ●  Using ThousandEyes we can see where a problem is brewing in the Internet and only direct

those users through Akamai This solution works better than anycast and FEC-only solutions.

16

Becoming independently reliable

We’re interested in techniques that do not involve cloud-based Internet reliability services and do not require changing the current nature of the Internet. Here are some of the ideas we’re beginning to explore. We look forward to getting more ideas from other experts in this area.

17

IXP IXP

Promote IXPs for business Internet

18

Business

Consumer Consumer

Consumer

Business

Consumer Consumer

Consumer

IXP

Business

private

Business Internet Internet

Access model Peering model

The trouble with the access model

ISP1

In the access model, since BGP cannot signal path quality, traffic is blindly steered over a path that has the shortest number of intermediate networks versus a path with the best performance. Buying extra access circuits to avoid Internet congestion is a fool's game.

BB Site ISP2

ISP3

ISP4

ISP5

IXPs

Internet

ISP1’s other customers

19

Access

Congested

Bloomberg Customers

Private

Troubled peering link as seen on ThousandEyes

20

Peering model advantage

Peering directly with major ISP at an IXP creates more direct access to a larger number of our Internet customer base at a better price point than buying access lines to numerous ISP. Peering with large businesses (many of our customers) at an IXP can also reduce costs and simplify business-to-business connectivity by integrating it with Internet connectivity.

21

ISP1

ISP2

ISP3

ISP4

ISP5

Internet

Congested

ISP1’s other customers

BB Site 1

BB Site 2

IXP2

IXP1

Peering

Bloomberg Customers

Who do the big guys peer with?

Here ThousandEyes gives us an idea how other major websites are managing their Internet presence. Internet topology becomes more visible as ThousandEyes deploys more monitors

22

Google Facebook

“Smart” DNS

Internet

A smart DNS is a distributed performance-aware DNS that directs users to the service location which will provide the best application experience to a user.

23

Multipath TCP (MPTCP)

Internet

Customer

Private

MPTCP takes advantage of ECMP opportunities along the path from one end of the Internet to the other. It can be configured to instantiate multiple sub-flows which can be hashed across multiple available paths. MPTCP automatically moves traffic on impaired subflows to good ones. End-to-end encryption required. Initially this can be implemented between corporate web proxies and load balancers. Bloomberg intends to support MPTCP standardization in Linux.

24

Are there really multiple usable paths?

This ThousandEyes graph shows the number of ECMP paths detected along three routes to a popular US-based web site.

Ping Traceroute

TCP Traceroute

25

Another graph of ECMP on Internet

26

Multi-site Multipathing

Internet

Site1

Site 2

Customer

Private

Customer site connects to service via application-level gateways located at multiple edge sites. This model requires no special network configuration but is generally proprietary. However it’s perfect for transactional protocols such as the FIX (Financial Information eXchange) protocol, adding both high availability and site-to-site security to applications that need both but come with neither.

27

Putting it all together - IXP peering ●  Avoid remote congested peering links ●  Connect “directly” at public meet-me points with other enterprises

Smart DNS ●  Direct users to the most performant edge site from their location

MPTCP ●  Leverage multiple intermediate paths ●  Shift traffic from impaired paths to good ones

Multi-site multipathing ●  Protect against failures to/at the edge site

28

talking point slides (as needed) 29

Network management at Bloomberg ●  Home-grown everything (almost). ThousandEyes is a rare exception. ●  Multi-vendor automation library and applications. ●  Network topology and inventory discovery using automation library. ●  Health checks using automation library. ●  Performance monitoring (4.8 million metrics per minute via SNMP). ●  Event promoting/demoting/coalescing/routing/alerting/ticketing. ●  Years of effort. Required seasoned network developers to get it right. ●  Technology evolving rapidly. Keeping up is a big challenge. Bloomberg is

not AT&T. Limited resources available for network development and management.

30

Network visibility -- control-plane

Out-of-band polling and streaming

●  SNMP, CLI, Syslog, APIs, etc. ●  Measure resource consumption and errors. ●  Does not measure experience. ●  Polling: Metrics/topology. Not possible for Internet. ●  Streaming: Metrics. Not possible for Internet. ●  Streaming: Topology. Possible to get AS-level topology of Internet via

BGP.

31

Network visibility -- data-plane

In-band probing ●  Measures experience. To really appreciate how bad a road is, you need

to drive it. ●  Best when paired with out-of-band data. ●  Types:

●  Over-the-top/outside-in: subject to blind spots unless numerous test points.

●  Hop-by-hop/inside-out: great for private networks, but need more vendors to open up. Not possible for Internet.

32

Types of network impairments Middle-mile impairment ●  Routing instability / screw-ups ●  Lossy or bottleneck links ●  Network latency

Last-mile impairment ●  Lossy or bottleneck links ●  DDoS / bad guys ●  Slow application (perceived by user as network impairment)

Locater impairment ●  DNS hijack / attacks ●  DNS instability / screw-ups

33