appsphere 15 - whipping your apps into shape: team beachbody shows you a devops performance workout...

Whipping Your Apps Into Shape Michael Lee, VP Technology

•  Introduction –  What is Beachbody? –  Our Growing Pain –  Our Problem

•  Decide. –  Why We Chose AppDynamics

•  Commit. –  Governance Plan –  Low barrier to entry: Make it accessible

and consumable •  Succeed.

–  Lean Out –  Explosive Speed –  Strength Training –  Our Solution

•  Q&A

What new features would you like to see in AppDynamics or features Beachbody should take a closer look at? Text the keyword BEACH and your response to 22333

$1.3B Grew exponentially 1998-2015 •  Fitness company with four major lines of business

–  Direct to consumer –  Network –  Certifications –  On Demand

•  Our programs –  1998 Turbo Jam –  2003 P90x –  2007 Team Beachbody –  2010 Shakeology –  2014 Beachbody Live –  2015 Apparel


Our challenge

•  Growth led to sprawl which led to knowledge isolation

•  Architecture was not ready for our increasing data and customer volumes –  Many key services SaaS hosted, with companies that did not grow with us

•  Bringing applications in house with a team more accustomed to hosted applications

•  Need to deliver user facing, web speed experience with marketing driven traffic spikes across eCommerce, CMS, digital distribution, and analytics –  Need to be able to monitor those businesses on more than just technical metrics in real time


Warehouse

Tax

Orders

Accounting

Payroll

Payments

Media

Customer Service

Procurement

Continuities

Telemarketing

Billing

1998

Text the keyword BEACH and your response to 22333

Warehouse

Tax

Orders

Call Switch

Accounting

Payroll

Payments

Media

Customer Service

Procurement

Continuities

Telemarketing

Billing

Community

eCommerce

1998 2000


Warehouse

Tax

Orders

Call Switch

Accounting

Payroll

Payments

Media

Customer Service

Procurement

Continuities

Telemarketing

Billing

Community

eCommerce

1998 2000 2005

Genealogy

Coach

Club

Commission

Content

Coach Reports

Chat

Affiliate


Warehouse

Tax

Orders

Call Switch

Accounting

Payroll

Payments

Media

Customer Service

Procurement

Continuities

Telemarketing

Billing

Community

eCommerce

1998 2000 2005 2010

Assets

Genealogy

Revenue

Revenue Recognition

Cash

Reporting

Accounts Payable

Coach

Club

Commission

Email Management

Content

Coach Reports

Coach Relations

General Ledger

Chat

Accounts Receivable

Replicated Sites

Affiliate


Warehouse

Tax

Orders

Call Switch

Accounting

Payroll

Payments

Media

Customer Service

Procurement

Continuities

Telemarketing

Billing

Community

eCommerce

1998 2000 2005 2010 2015

Assets

Genealogy Order Reporting

Properties

Marketing Database

Projects

Revenue

Revenue Recognition

Video

Workshops

Cash

Reporting

Accounts Payable

Coach

Club

Commission

Email Management

Fitness Tracking

Content

Coach Reports

Coach Relations

General Ledger

Chat

Accounts Receivable

Replicated Sites

Affiliate Email Marketing


Warehouse

Tax

Orders

Call Switch

Accounting

Payroll

Payments

Media

Customer Service

Procurement

Continuities

Telemarketing

Billing

Community

eCommerce

1998 2000 2005 2010 2015 2016+

Assets

Genealogy Order Reporting

Properties

Marketing Database

Projects

Revenue

Revenue Recognition

Video

Workshops

Cash

Reporting

Accounts Payable

Coach

Club

Commission

Email Management

Fitness Tracking

Content

Coach Reports

Coach Relations

General Ledger

Chat

Accounts Receivable

Replicated Sites

Affiliate Email Marketing

International


i3c

Media

Quickbooks

Warehouse Telemarketing

Initial Years


Four Resources in IT Basic Accounting <3 3rd Party Integrations

i3c

Media

Quickbooks

Warehouse Telemarketing eCommerce

Call Switch

Message Boards

Customer Service eCometry

mybeachbody

Enterprise Service Bus

Growing Pains


3rd Party Partner Integrations 150 Resources in IT

i3c

Media

Quickbooks

Warehouse Telemarketing eCommerce

Call Switch

Message Boards

Customer Service eCometry

mybeachbody Coach CRM Certification Digital

Enterprise Service Bus

Today


3rd Party Partners Critical to our business 300+ Resources in IT

Media Agencies

Channels & Affiliates

Dist Center (3PL) Telemarketing Manufacturer Supplier B2B

Countries COO BB in a Box BB Live bb.com

External Service Layer

CRM Email Delivery

Content Management MDM

Commerce Engine

ERP

MLM Engine

Non-Transactional

Corporate Systems

Data Asset Management

Media Mgmt (Pioneer)

Merchant Portal

Campaign & Test

Management BI Portal C/S Portal R&D

(Agile PLM)

Payment Service

Internal Service Layer

Streaming Players

Club.tbb.com

2018

Micro-service Architecture API –driven Ready for International Exponential Revenue Growth Target Partner Development Centers

Obese Too many tools, platforms, and services. PCI Compliance, SoX, Security, & International Growth

Slow MTTR on $150,000/hour. Business growing and we couldn’t keep up

Weak 3rd Party Integrations were owning us. We were losing focus on the user.

Our Problem: No Foundation. No Anchor. No DevOps Culture


Decide. Choose your program.


Why we chose AppDynamics

End-to-end visibility, from NOC to Engineer to Product Manager

•  We have a team focused on integration with a large enterprise software footprint –  Wanted instant value without code changes

•  End user monitoring was key to our strategy of providing higher levels of service to our users –  Bringing monitoring beyond DevOps all the way to Business Value

•  Integrated triage –  Minimize cultural impact while rolling out DevOps to a company unaccustomed to the level of responsibility

•  Partnership and support –  Software with Service, not just SaaS


Commit. Implement the solution.


Drive Engagement

Culture = Values + Execution

•  Training

•  Governance plan –  Two TAM’s + a project manager coordinating with our internal teams –  Across Eight Environments –  One Dev, one Dev/Int, three QA, one Staging, one UAT, one Production

•  Innovation Hackday –  iPad to the winner of the AppD prize – Release Engineer automated

rollout to our systems

•  DevOps contest –  7 Teams –  15 Individuals

•  Beachbody takeover of LA AppDynamics User Group –  Over 50 People show up


So, what are we monitoring? What apps? What databases? What sources? What scale?

1,600

250 0

400

800

1200

1600

2000


Succeed. The “after” photo.


Copyright © 2015 AppDynamics. All rights reserved. 22


Governance & DevOps Culture

Lean Out. Focus & Anchor on the Core User

Text the keyword BEACH and

your response to 22333

ADOPT 6. AppDynamics



BB Live Dashboard (Eng/Product) Business Metrics Tech Metrics

Product Metrics

Responsive

Desktop



Executive Dashboard



Explosive Speed. Improve Goodwill and Increase Revenue

Leverage Advanced AppDynamics. DevOps Solutions Must Empower The Delivery Team



Automatic Application Mapping



That’s not all…. + =

So, what does this really mean? 1.  AppD detects MySQL is down 2.  It clears out the bad bin files 3.  It restarts MySQL 4.  You know it happened, but nobody had to wake up!

Automated Remediation


Automated Remediation

DevOps + AppDynamics = a better night’s sleep

•  Imagine keeping the entire application relationship in your mind •  How do you when the application slowness you’re seeing is your

app or a lower tier? •  With dynamic mapping, you can see how the apps are related,

where slowness in one app is propagating up •  This allows you to scale up the correct tier, focus on indices in

the correct DB, and not waste your time in examining a victim rather than the perpetrator

•  You can have your NOC call the right team to fix the underlying problem, rather than waking somebody up to triage a symptom



War Room

•  Beachbody is heavily dependent on consulting and third parties. •  Driving consensus and a consolidated view of what’s interesting

during triage is key to resolving a problem •  War room allows for a single channel to bring people on

disparate systems together looking at the same data, talking in the same place.

•  It’s a great read-only place to share data with third parties you’re troubleshooting with –  Show them yours through the built-in widgets

•  Pull in theirs using iFrames



What’s your normal bridge look like?


Who’s on the bridge?

What’s been

done? Who’s on the bridge?

What’s been

done?


What’s been

done?

What’s been

done?



Wouldn’t this be better….?


What’s been done?

What logs are pertinent?

What’s our checklist?

What’ve we looked at so far?



Strength Training. Pack the Muscle. Hold Your Partner Accountable.

Trust but Verify. Measure, Monitor, and Test what you Manage.

End User Monitoring

•  Auto-baselining for: –  Requests per Minute –  Errors per Minute –  Average Response Time per minute

•  Coupled with Health Rules for: –  One standard deviation from the baseline –  Replaces [Insert Your Synthetic Monitoring Here] with something that

has fewer false positives •  Why use synthetics then?

–  For sites with little traffic or very spiky traffic -- as a secondary complement to EUM, rather than the other way around



Good news about Synthetics

We’ve been using the beta, working closely with AppD, and it’s getting there. •  The good:

–  Seeing it alongside our existing metrics –  Using it as complement to EUM on our newer sites

•  What we’re excited about using: –  Script based testing

•  Vanilla Python environment –  Automatic retest on failure

•  No more guessing if it’s the node or the test



EUM vs. Synthetics


Test Locations Reflective of your actual users, and scales as your users scale

Test Timing Near-real-time based on actual users

Works with low traffic and 3rd party sites

You need enough consistent traffic to baseline; must be on your system

False Positives Less frequent with sites with enough traffic

Not necessarily reflective of where your end users are located

Scheduled intervals

Not traffic or hosting dependent

Dependent on whether individual nodes are working

EUM Synthetics

Complementary strengths and roles


Accountability of 3rd Parties

•  Synthetic monitoring provides a consistent measure of the performance of all 3rd party services


Identify deviations from normal performance using consistent measurements

Obese Too many tools, platforms, and services. PCI Compliance, SoX, Security, International

Slow MTTR on $150,000/hour. Business growing and we couldn’t keep up

Weak 3rd Party Integrations were owning us. We were losing focus on the user.

Our Problem: No Foundation. No Anchor. No DevOps Culture




Our Solution: Core. Anchor. DevOps Culture.

Lean Out Focus on the Core User. Governance & DevOps Culture Result: Maximized our Investment.

Explosive Speed

Leverage Advanced AppDynamics. DevOps Solutions Must Empower The Delivery Team Result: MTTR down from hours to minutes

Strength Training

Trust but Verify. Measure, Monitor, and Test what you Manage. Result: 3rd Party Vendors/Partners Improved Stability

Shakeology



Governance & Culture

Dashboards Automated Remediation & Automatic App

Mapping

War Room End User Monitoring

Synthetics

Questions?


Answer a Question. Win a Program!


1. How many AppD features are you using? 2. How many systems are you monitoring? 3. What have you found works in your dashboard? What gets

your stakeholders engaged? 4. How are you using APM or DevOps to support a major

business goal that IT has to deliver? 5. What is your favorite AppD feature and why?


Appendix


The results: un-Photoshopped A war isn’t won in a single battle. •  We reached Engineering and Product

–  Drink our own champagne and start looking at our products more deeply –  This is the dashboard we’ll see on slide 13:

•  https://beachbody.saas.appdynamics.com/controller/#/location=CDASHBOARD_DETAIL&timeRange=last_12_hours.BEFORE_NOW.-1.-1.720&mode=MODE_DASHBOARD&dashboard=47

•  We reached executives –  Show the power of near-real-time analytics vs. 24hrs to upload analytics data –  This the dashboard we’ll see on slide 14:

•  [Insert link to executive dashboard] •  We reached Ops

–  Give our Ops team a chance to sleep at night –  This is the workflow we’ll see on slide 14:

•  [Insert link here to automated triage flow] Copyright © 2015 AppDynamics. All rights reserved. 46

Our footprint


Server side

Java

PHP

Node

MySQL (Percona)

Applications Databases

More than 1600 nodes, including 200+ databases

Client side

EUM

Synthetics

Oracle

Isolate test where errors were verified


Compare content across geography and time


appsphere 15 - whipping your apps into shape: team beachbody shows you a devops performance workout...

Documents