lessons learned from verizon: implementing microservices with go90
TRANSCRIPT
![Page 1: Lessons Learned from Verizon: Implementing Microservices with go90](https://reader030.vdocuments.net/reader030/viewer/2022032711/587332001a28ab596c8b6ca1/html5/thumbnails/1.jpg)
How Verizon's go90 streams the NFL Superbowl with Lightbend
TechnologiesChristopher Webster
Verizon / AOL
![Page 2: Lessons Learned from Verizon: Implementing Microservices with go90](https://reader030.vdocuments.net/reader030/viewer/2022032711/587332001a28ab596c8b6ca1/html5/thumbnails/2.jpg)
Outline● Overview of go90
● Service Architecture
● Traffic Spikes
● Availability
● Caching
![Page 3: Lessons Learned from Verizon: Implementing Microservices with go90](https://reader030.vdocuments.net/reader030/viewer/2022032711/587332001a28ab596c8b6ca1/html5/thumbnails/3.jpg)
Go90 App
![Page 4: Lessons Learned from Verizon: Implementing Microservices with go90](https://reader030.vdocuments.net/reader030/viewer/2022032711/587332001a28ab596c8b6ca1/html5/thumbnails/4.jpg)
What is go90● App
○ Mobile app supporting both video on demand and live event streaming
● Platform
○ APIs for app and website delivery
■ Signup / Registration
■ Video content metadata
■ Social interactions
● Comments / sharing / favorite
○ Management APIs
■ Content restrictions
■ Content metadata - automatic ingestion
■ Merchandising
![Page 5: Lessons Learned from Verizon: Implementing Microservices with go90](https://reader030.vdocuments.net/reader030/viewer/2022032711/587332001a28ab596c8b6ca1/html5/thumbnails/5.jpg)
Client
CDN/WAF
Edge API
Accounts
SocialNotifications
ExperimentsSearch
Discovery2D View
Metadata
![Page 6: Lessons Learned from Verizon: Implementing Microservices with go90](https://reader030.vdocuments.net/reader030/viewer/2022032711/587332001a28ab596c8b6ca1/html5/thumbnails/6.jpg)
Technologies● The edge services all use Scala (2.11) with Java 8
● Cassandra, Elastic Search, Kafka, and some Mongo
● Cloud native application
● Play web services 2.3 (2.4 migration in progress)
![Page 7: Lessons Learned from Verizon: Implementing Microservices with go90](https://reader030.vdocuments.net/reader030/viewer/2022032711/587332001a28ab596c8b6ca1/html5/thumbnails/7.jpg)
Auth
Find Content
Landing Page
End
Get Key Play Content
Video app during a live event
![Page 8: Lessons Learned from Verizon: Implementing Microservices with go90](https://reader030.vdocuments.net/reader030/viewer/2022032711/587332001a28ab596c8b6ca1/html5/thumbnails/8.jpg)
Traffic Spikes
10x
4 min
![Page 9: Lessons Learned from Verizon: Implementing Microservices with go90](https://reader030.vdocuments.net/reader030/viewer/2022032711/587332001a28ab596c8b6ca1/html5/thumbnails/9.jpg)
Traffic Spikes● Timed Scaling
○ Can be used for known events
○ Difficult to predict the actual load, could still be under provisioned
● Autoscaling
○ Helps but takes time
○ Adjust the policies to assume a spike (aggressively adding capacity)
○ Adding capacity around the utilization target
● Load Shedding with Circuit Breaker allows popular experience to be preserved
○ Failure triggers default behavior (serving static initial experience page)
● Use CDN where appropriate in extreme cases
![Page 10: Lessons Learned from Verizon: Implementing Microservices with go90](https://reader030.vdocuments.net/reader030/viewer/2022032711/587332001a28ab596c8b6ca1/html5/thumbnails/10.jpg)
Availability● Microservices help partition outages to specific areas
○ Failure of user login wouldn’t impact video data
● Extensive use of Future recover
● Use of Akka CircuitBreaker in conjunction with failure recovery to further aid
○ Fail requests fast and use recovery if downstream services are in trouble
● Employing load shedding techniques
○ Upstream techniques but also use a filter to keep track of number of requests in flight
■ Shed load as the normal limit is exceeded
● Bounded queues
○ Service calls, database persistence queues
![Page 11: Lessons Learned from Verizon: Implementing Microservices with go90](https://reader030.vdocuments.net/reader030/viewer/2022032711/587332001a28ab596c8b6ca1/html5/thumbnails/11.jpg)
Future Recovery Example
![Page 12: Lessons Learned from Verizon: Implementing Microservices with go90](https://reader030.vdocuments.net/reader030/viewer/2022032711/587332001a28ab596c8b6ca1/html5/thumbnails/12.jpg)
Circuit Breaker Example
![Page 13: Lessons Learned from Verizon: Implementing Microservices with go90](https://reader030.vdocuments.net/reader030/viewer/2022032711/587332001a28ab596c8b6ca1/html5/thumbnails/13.jpg)
Caching - Hints● CDN / app caching help with extreme failures
● Use prewarmed caches
● Use caution around automatic cache expiration
● Separate hot cache from on demand cache
![Page 14: Lessons Learned from Verizon: Implementing Microservices with go90](https://reader030.vdocuments.net/reader030/viewer/2022032711/587332001a28ab596c8b6ca1/html5/thumbnails/14.jpg)
Questions