architecting for scale - cs.cmu.educkaestne/15313/2017/... · source: josh evans, mastering chaos...

10/16/17

1

Architecting for ScaleMicroservices at Netflix-Uber-Spotify Scale

PooyanJamshidi

FoundationsofSoftwareEngineering LearningGoals

• Understandthevalueofmicroservices forbuildingcomplexapplicationsthatneedtooperateathigherscale• Identifyrequirementsthatderivecompaniestomigratetomicroservices (contrastofrequirementsbetweencompanies)• Understandstrategiesforreliability of microservice architectureeitheratmicro-levelusingdesignpatternsoratalargerlevel• Buildagileteamstructurethatenablelarge-scalecompaniestomovefast(organizationalchallenges)• UnderstandchallengesthatNetflix-Uber-Spotifyfacedinrealizingmicroservice basedapplications

2

Disclaimer

• Iusedmaterialsfrom• Netflixblog• Spotify,UberandNetflix’sarchitectsGOTOtalks• Andsomeothersourcesreferencedintheslides

• I’mapostdocinChristian’sgroup• SoftwareEngineering+MachineLearning

• Iworkedasasoftwarepractitionersfor7years• Pre-PhD• 4yearsasadeveloper• 3yearsasanarchitect• Involvedinmigrationtocloudandmicroservices

2 IEEE SOFTWARE | PUBLISHED BY THE IEEE COMPUTER SOCIETY 0 7 4 0 - 7 4 5 9 / 1 6 / $ 3 3 . 0 0 © 2 0 1 6 I E E E

Microservices Architecture Enables DevOpsMigration to a Cloud-Native Architecture

Armin Balalaie and Abbas Heydarnoori, Sharif University of Technology

Pooyan Jamshidi, Imperial College London

// This article reports on experiences and lessons learned during incremental migration and architectural refactoring of a commercial mobile back end as a service to microservices architecture. It explains how the researchers adopted DevOps and how this facilitated a smooth migration. //

A LOOK AT the searches related to the term “microservices” on Google Trends revealed that the top searches are now technology driven. This im-plies that the time of general search terms such as “What is microser-vices?” has now long passed. Not only are software vendors (for ex-ample, IBM and Microsoft) using microservices and DevOps practices,

but also content providers (for exam-ple, Netflix and the BBC) have ad-opted and are using them.

In addition, Google Trends re-veals that both DevOps and mi-croservices are growing concepts, with an equal rate of growth after 2014 (see Figure 1). Although Dev-Ops can also be applied to mono-lithic software systems, microservices

enable effective implementation of DevOps by promoting the impor-tance of small teams.1 (For more on DevOps and Microservices, see the related sidebar.)

A microservices architecture is a cloud-native architecture that aims to realize software systems as a package of small services. Each ser-vice is independently deployable on a potentially different platform and technological stack. It can run in its own process while communicat-ing through lightweight mechanisms such as RESTful or RPC-based APIs—for example, Finagle. (REST stands for Representational State Transfer.) In this setting, each ser-vice is a business capability that can utilize various programming lan-guages and data stores and is devel-oped by a small team.2

Migrating monolithic architec-tures to microservices brings in many benefits. In particular, it pro-vides adaptability to technological changes to avoid technology lock-in and, more important, reduced time-to-market and better development team structuring around services.3

Here we explain our experiences and lessons learned during incre-mental migration of Backtory (www.backtory.com), a commercial mo-bile back end as a service (MBaaS), to microservices in the context of DevOps. Microservices help Back-tory in various ways, especially in shipping new features more fre-quently and providing scalability for the collective set of users from differ-ent mobile-app developers.

Furthermore, we report on migra-tion patterns we developed on the basis of our observations in migra-tion projects. Practitioners can use these patterns to migrate monolithic software systems to microservices. In addition, system consultants can use

FOCUS: DEVOPS

WhatisthemostinterestingaspectthatyouhavelearnedfromtheNetflixtalk?

Tradeoffinsoftwarearchitecture

• Everythingistradeoff• Trytomakethemintentionally

OrganSystemsEachorganhasapurposeOrgansformsystemsSystemsformanorganism

10/16/17

2

ELB

andsoistakingtraffic

Source:JoshEvans,MasteringChaos- ANetflixGuidetoMicroservices

LargestInternetTVnetwork

86millionmembers~190countries,10soflanguages125mhourscontentperday

Microservices onAWS


NetflixDVDDataCenter- 2000

LinuxHostApache Tomcat

JavawebSTORE

Load

Balan

cer

BILLING

HTTP

JDBC

DBLinkHTTP/S

MonolithiccodebaseMonolithicdatabaseTightlycoupledarchitecture

Whatmicroservices arenot Whatmicroservices arenot

Service Oriented Architecture

Enterprise Service Bus (ESB)

PrivacyService (PS)

ServiceConsumer

ServiceConsumer

ServiceConsumer

ServiceProvider

ServiceProvider

ServiceProvider

OtherServices

Routing

Service Registry Transport

Replication

• Message Routing • Message Monitoring

• Service Replication • Language transformation

Edge

ELB

Zuul

NCCP

API

MiddleTier&Platform

Product• Buckettesting• Subscriber• Recommendations

Platform• Routing• Configuration• Crypto

Persistence• Cache• Database


Architecturalpattern1:APIGateway

12Source:Kasun Indrasiri,Microservices inPractice:FromArchitecturetoDeployment

10/16/17

3

Architecturalpattern2:Inter-processCommunicationinaMicroservices Architecture

13

Architecturalpattern3:ServiceDiscoveryinaMicroservices Architecture

14

Architecturalpattern4:Event-DrivenDataManagementforMicroservices

15

27Microservices – From Design to Deployment Ch. 3 – Inter-Process Communication

A message consists of headers (metadata such as the sender) and a message body. Messages are exchanged over channels. Any number of producers can send messages to a channel. Similarly, any number of consumers can receive messages from a channel. There are two kinds of channels, point-to-point and publish-subscribe:

• A point-to-point channel delivers a message to exactly one of the consumers that are reading from the channel. Services use point-to-point channels for the one-to-one interaction styles described earlier

• A publish-subscribe channel delivers each message to all of the attached consumers. Services use publish-subscribe channels for the one-to-many interaction styles described above

Figure 3-4 shows how the taxi-hailing application might use publish-subscribe channels

Figure 3-4. Using publish-subscribe channels in a taxi-hailing application.

The Trip Management service notifies interested services, such as the Dispatcher, about a new Trip by writing a Trip Created message to a publish-subscribe channel The Dispatcher finds an available driver and notifies other services by writing a Driver Proposed message to a publish-subscribe channel

DISPATCHER

TRIP CREATED

DRIVER PROPOSED

TRIPMANAGEMENT

PASSENGERMANAGEMENT

DRIVERMANAGEMENT

ChrisRichadson,Microservices fromDesigntoDeployment

Architecturalpattern5:DecentralizedDataManagement

ChristianPosta,TheHardestPartAboutMicroservices:YourData

Architecturalpattern6:ChoosingaMicroservices DeploymentStrategy

17

Architecturalpattern7:Security

18Source:Kasun Indrasiri,Microservices inPractice:FromArchitecturetoDeployment

10/16/17

4

Whatarchitecturalpatternsyoucanidentifywithinthesearchitectures?

Edge

ELB

Zuul

NCCP

API

MiddleTier&Platform

Product• Buckettesting• Subscriber• Recommendations

Platform• Routing• Configuration• Crypto

Persistence• Cache• Database


Microservices atUber

21

MicroservicesatSpotify

22

Whyreliabilitymattersinmicroservices world?

LinuxHostLinuxHost

LinuxHostLinuxHost

Intra-serviceRequests



Networklatency,congestion,failureLogicalorscalingfailure

ServiceA ServiceB


10/16/17

5

CrossingtheChasm


CascadingFailure


Vaccination

Device ServiceB

ServiceC

Internet EdgeZuul

ServiceA

ELB

FITSynthetictransactionsOverridebydeviceoraccount%oflivetrafficupto100%

FaultInjectionTesting(FIT)


Device ServiceB

ServiceC

Internet EdgeZuul

ServiceA

ELB

FIT

FaultInjectionTesting(FIT)

Enforcedthroughoutthecallpath


10/16/17

6

ELB

APIAPI

Gateway

App1

App2

App4

App5

App6

App3

App7

App8

99.99

99.99

99.99

99.99

99.99

99.99

99.99

99.99Proxy

99.99 99.99

CombinatorialMath

99.9910 =99.9

CriticalMicroservices

Persistence

Inthepresenceofanetworkpartition,youmustchoosebetweenconsistencyandavailability

CAPTheorem

DB

DB

DB

NetworkB

NetworkC

NetworkD

Service

NetworkA X ZoneA

ZoneB

ZoneC

ZoneB

ZoneC

Client

ZoneA

LocalQuorum(Typical)

100ms

EventualConsistency

10/16/17

7

Infrastructure

December24th,2012

US-East-1

Canada

Noplacetogo

US

LatinAmerica

US-East-1US-West-2 EU-West-1

Regionalfailover

x x

Regional fail-over

Ruslan Meshenberg,Microservices atNetflixScale:Principles,Tradeoffs&LessonsLearned

RegionalfailoverRegional fail-over


10/16/17

8

Whatisastatelessservice?


• Notacacheoradatabase• Frequentlyaccessedmetadata• Noinstanceaffinity• Lossanodeisanon-event


Minimumsize

Desiredcapacity

Maximumsize

Scaleoutasneeded

S3AMIretrievedondemand

ComputeefficiencyNodefailureTrafficspikesPerformancebugs

AutoScalingGroups

ClusterA ClusterD

EdgeCluster

ClusterB

ClusterC

Surviving Instance Failure


Whatisastateful service?

10/16/17

9


• Databasesandcaches• Customappswhichholddata• Lossofanodeisanotableevent

DedicatedShards– AnAntipattern

Squid1 Squid2 Squid3

ClientApplication

SubscriberClientLibrary

CacheClient ServiceClient

S S S S...

DB DB DB DB...

Squidn

HAProxy

Set1 Set2 Set3 Setn

X

Redundancyisfundamental

ZoneA ZoneB ZoneC

.........

EVCache Writes

ClientApplication

ClientLibrary

EVCacheClient

ClientApplication

ClientLibrary

EVCacheClient

ClientApplication

ClientLibrary

EVCacheClient

...

ZoneA ZoneB ZoneC

ClientApplication

ClientLibrary

EVCacheClient

.........

EVCache Reads

ClientApplication

ClientLibrary

EVCacheClient

ClientApplication

ClientLibrary

EVCacheClient

...

Whyautomation,inallsoftwaredev/opsstages,isimportant?

10/16/17

10

AutonomicNervousSystem

Youdon’thavetothinkaboutdigestionorbreathing

PrioritiesOur Priorities

1. Innovation

3. Efficiency

2. Reliability


Innovation:Tightcouplingdoesn’twork Monolithicvsmicroservice-basedapplications:InterdependentvsIndependentteams

59LeeAtchison,ArchitectingforScale:HighAvailabilityforYourGrowingApplications

End-to-endownership

60

End-end ownership + velocity Architect

Design

Develop

Review Test

Deploy

Run

Support

Architect

Design

Develop

Review Test

Deploy

Run

Support

Architect

Design

Develop

Review Test

Deploy

Run

Support

Architect

Design

Develop

Review Test

Deploy

Run

Support

Architect

Design

Develop

Review Test

Deploy

Run

Support

Architect

Design

Develop

Review Test

Deploy

Run

Support

Architect

Design

Develop

Review Test

Deploy

Run

Support

Architect

Design

Develop

Review Test

Deploy

Run

Support

Architect

Design

Develop

Review Test

Deploy

Run

Support

Architect

Design

Develop

Review Test

Deploy

Run

Support

Architect

Design

Develop

Review Test

Deploy

Run

Support

Architect

Design

Develop

Review Test

Deploy

Run

Support

Ruslan Meshenberg,Microservices atNetflixScale:Principles,Tradeoffs&LessonsLearned KevinGoldsmith,Microservices @Spotify

10/16/17

11

Server

Core Library

Platform Platform Platform Platform

Infrastructure

KevinGoldsmith,Microservices @Spotify

ChallengesSynchronization

Client UX implementationCore Library Implementation

depends on depends on depends onServer Implementation

Infrastructure Implementation



Full-stack autonomous teamsRequires you to structure your application in loosely coupled parts

ArchitectureevolutionofSpotify

67KevinGoldsmith,Microservices @Spotify

10/16/17

12

ArchitectureevolutionofSpotify

68 69

70 71

Load

Bal

lanc

er

72

Microservices:Yay!

• EasiertoScale• Easiertotest• Easiertodeploy• Easiertomonitor• Theycanversionedindependently

10/16/17

13

Microservices:Boo!

• Monitoringlotsofservices• Documentations• Increasedlatency

75

What does this look like at Spotify?

‣ 810 active services

‣ ~10 Systems per squad

‣ ~1.7 Systems per person with access to production servers

‣ ~1.15 Systems per member of Technology

MicroservicesatSpotify

76

As of April 2016:

Uber Cities Worldwide: 400+ Countries: 70 Employees: 6,000+

Uber

MattRanney,WhatIWishIHadKnownBeforeScalingUberto1000Services

Microservices atUber

79MattRanney,WhatIWishIHadKnownBeforeScalingUberto1000Services

10/16/17

14

pre-history PHP (outsourced)

Dispatch Node.JS, moving Go

Core Services Python, moving to Go

Maps Python and Java

Data Python and Java

Metrics Go


LANGUAGESHard to share code Hard to move between teams WIWIK: Fragments the culture


MattRanney,WhatIWishIHadKnownBeforeScalingUberto1000Services MattRanney,WhatIWishIHadKnownBeforeScalingUberto1000Services

Summary

• Microservices maybearightsolutionforbuildingcomplexapplicationsthatneedtooperateathigherscale• Tradeoffsthatcompaniesmadetomigratetomicroservices (contrastofrequirementsbetweencompanies)• Makingreliablemicroservice architecturerequiresstrategiestodealwithfailureeitheratmicro-leveloratalargerlevel• Microservices architecturehelptobuildagileteamstructurethatenablelargescalecompaniestomovefast(organizationalchallenges)

architecting for scale - cs.cmu.educkaestne/15313/2017/... · source: josh evans, mastering chaos...

Documents