MLB Advanced MediaCustomer Driven DevOps at Work:

Operating the Digital Turnstile

FutureStack ’16 - 09 Aug 2016 Sean Curtis, SVP Software Engineering

Agenda• MLBAM - Brief Intro

• The New Digital Turnstile

• The Evolution of DevOps at MLBAM

• Our Platform

• Our Customers

• NewRelic Helping Power Customer-Driven DevOps

MLBAM - Who we are• Formed in 2000 to centralize all digital media

distribution on the Internet

• Over 1200 employees, predominantly in NY, San Francisco

• Over half of our staff are in engineering, systems, and operations

• Operate all digital functions for 30 MLB teams, as well as MLB.com league portal

The Beginning• 2002 - First sports league to live stream on the

internet.

• 30K viewers

• One Device platform: PC

• 280 Kbps

• 244x182

Where we are now…

• 2016 - #1 OTT Sports Streaming Service

• 400+ Devices

• 4.5 Mbps

• 720p HD

The Digital TurnstileAn Evolution

Turnstiles: Early Stage• One ticket source (theatre)

would sell exclusive tickets to get into a show

• The “bigger” or more popular the movie, the longer the line for access.

• Customer interest inversely correlated to Customer experience…

Friction!

Evolution• Schedule information made widely

available. Customers have greater choice in location and showing time

• Purchase tickets ahead of time from the box office

• Multiple forms of payment or billing.• Purchase tickets online from

multiple sources. Pay ahead of time.

• Pick your particular showing and save your seat!

• The system now scales better to meet customer demand.

The Digital Turnstile

video, events, data,

media rights

user, location, device,

App Store, etc.

DevOps at MLBAM

MLBAM DevOps: A Timeline

2001 2013

MLBAM, like many other Dev shops, has a “silo-ed” approach to

build vs. deploy / test & monitor

2016

MLBAM moves toward DevOps using tools for

build/deploy, configuration and

monitoring

2014

“Silo”-ed Ownership• Engineers build and test locally, then check in

artifacts to shared repository. Developers then document (when possible) through release notes & run books

• Systems actually does deployment, installing artifacts into test environments

• Developers/Project Managers then publish release to QA to coordinate testing. QA reports findings, eventually signing off on release. Load tests ensue, etc.

• If all testing passes, release to production by Systems.

• Operations now owns first-tier customer response.

“Silo”-ed Ownership

Dev

Systems/QA

Operations

Challenges we faced• Who actually owns the customer?

• How do we drive transparency (e.g. ubiquitous metrics) across all tiers?

• Does anybody or everybody have deep knowledge of the overall architecture?

• How do we properly escalate issues and priority between tiers?

• What portion of the system has trouble scaling as business demands more aggressive timelines?

A Shift in Architecture (and more)

Change in Thinking• MLBAM becomes a trusted partner to many large-scale

sports leagues and media brands (just to name a few)

Leverage proven solutions

• Spend more time on the things that matter to your customers.

• Rely on vendors for building / scaling infrastructure

• Opportunity to reimagine our existing systems while iterating on scalability, security and resiliency

MLBAM DevOps• MLBAM also formalizes an internal team, “Core

Engineering” to productize internal development tracks such as: toolset selection, configuration management, and infrastructure deployment

• Core Engineering engages with customer-facing teams to help “on-board” them to these various products and tools.

• Also helps spread best practices, architectural reviews, lessons learned across all dev teams

Engineers are motivated by…

• Being challenged to learn a new set of skills

• Owning more of the customer facing solution

• Paying down technical debt and making their products and services more scalable and resilient

• Building with transparency

Systems & Operations folks enjoy…

• Crisp architecture with time-tested methods for scaling (bonus points for automation!)

• Redundancy - enough said.

• An explicit networking and security model (and someone who understands ipchains)

• Automated tools for building and configuring infrastructure. (Bonus points for VCS integration!)

• A clear escalation path and chain-of-custody model for customer-facing issues

QA folks want…• The ability to spin up testing resources at a

moments notice

• Deeper metrics that describe system performance as a function of a recent change

• Sure-fire way of ensuring that iterations to code indeed satisfy the original requirements in all phases of testing

Business leaders want…

• Ubiquitous set of metrics that describe customer response and / or impact

• If / when issues arise, the fastest path to root cause and resolution for the customer

• An approach to scaling that aligns with business growth

Our Platform

+

Platform: AWS• Refactor hosted infrastructure (JMS Brokers, etc.) for

SNS, SQS - Improved scaling and resiliency

• Leverage IAM and other AWS products for access control, security, auditing, white/blacklist, networking, etc.

• Implement autoscaling of microservices for resource-level granularity of usage and cost

• Manage TCO per microservice, understand overprovisioning at the instance and/or product layer

Platform: NewRelic• Leverage full NewRelic platform across all tiers (mobile,

web, service)

• Foster collaboration between operations, systems, engineering, QA, and business teams by providing dashboards with ubiquitous metrics

• Establish fine-grained tolerance thresholds per metric observed to define business-critical incidents

• Integrate incident creation with other internal tools to identify, route, triage, resolve, and provide postmortem metrics

The New Turnstile

video, events, data,

media rights

user, location, device,

App Store, etc.

Our Customers• Similar to the movie ticketing example, MLBAM

has partnered with many third parties to provide unprecedented selection in how baseball fans can access our teams and league

Turnstile complexity• Authenticating users (AuthN)

and Authorizing them for access (AuthZ) become distributed operations

• In some cases, MLBAM is record holder, but in many cases, third parties sell MLB.tv to their subscribers (e.g. IOS, Pay-TV companies)

Turnstile complexity• We have to scale our turnstile

in order to meet peak demand (i.e. event start time) which differs dramatically between events

• Partner response times are not always equal or predictable

• From time to time, partners will have outages, both scheduled and unscheduled

How does New Relic help here?

• APM - provides implicit and explicit application performance monitoring

• Helps describe the overall health of each of our services

• Will create incidents and send them to downstream alerting/monitoring systems

• Drives more rapid and accurate insight as to root cause

• Provides single “pane of glass” for Ops, Systems, and Engineers at BAM

A brief example• The Rangers and Penguins are

playing in a late April NHL matchup that has playoff implications

• Hockey fans are trying to join the game live

• If any of our systems, or our partner systems has a hiccup, how can we quickly isolate any issues and take mitigation steps?

A Happy Turnstile (Green)

Operational Effects• NewRelic has become a default standard method of

providing simplified views of the state of our infrastructure

• Engineering now first class citizen in operational aspects of running our services, vs code/document-handoff, and have a greater understanding because they own infrastructure as well.

• SysOps now has fullly transparent view of the health of each service, but more importantly, the relationships between services and their dependencies

• Overlapping ownership of the customer!

Overlapping Ownership

Engineering Systems/QA Operations Business

Thank You!