evolving operations for bbc iplayer

of 23/23
Evolving Operations for BBC iPlayer Robert Shield Principal Software Developer, BBC Media Services

Post on 16-Apr-2017

2.733 views

Category:

Technology

1 download

Embed Size (px)

TRANSCRIPT

  • Evolving Operations for BBC iPlayer

    Robert ShieldPrincipal Software Developer, BBC Media Services

  • Introduction

  • Traditional capital investment model Monolithic codebase Centralised relational database Few releases per year Dedicated Ops team Storage!

    Video Factory - Before

  • Many small components Stateless & scalable AWS CloudFormation Continuous Delivery New components simple to deliver Storage!

    Video Factory - Today

  • SD VIDEO 1.3 mb / sec / channel 21 channels 2.3 TB per day

    Video Factory Some Numbers

  • SD VIDEO 1.3 mb / sec / channel 21 channels 2.3 TB per day

    Video Factory Some NumbersHD VIDEO 4.2 mb / sec / channel 10 channels 3.6 TB per day

  • 6 TB per day

    Video Factory Some Numbers

  • 6 TB per day

    Video Factory Some Numbers

    In 2 locations

  • 6 TB per day

    Video Factory Some Numbers

    In 2 locations

    With 2 copies in each location

  • 24TBper day

  • COSMOS

    Hand spunDeployment managerOne touch deploymentUses AWS CloudFormation on a per component basisGUI & API

    Video Factory Tools

  • GON

    SQS front endView on multiple queuesDrain, re-spool & download

    Video Factory Tools

  • Jenkins

    Continuous IntegrationUnit & BDD TestingKick off Cosmos deploys to Integration

    Video Factory Tools

  • Daskboard

    Developer ledMany different viewsUses API of other toolsConstantly updated

    Video Factory Tools

  • CloudFormation abstraction Stack fetcher / updater iSpy Splunk Chaos Monkey from @NetflixOSS

    Video Factory Tools

  • Splunk

    All components iSpyView pulse of workflowConstantly evolving views on data

    Video Factory Monitoring

  • Simulcast Console

    EC2 instanceFed by 4 componentsManagement capabilities

    Video Factory Monitoring

  • Zenoss

    Traditional monitoringNow available direct via SQSTied in to Amazon CloudWatch & Splunk

    Video Factory Monitoring

  • Integration between devs & ops More supportable Documentation Standardisation & automation Smaller releases, more often Increased buy-in

    Video Factory - Operations

  • Single region dependency Single account limits 3rd party software SDKs Monitoring

    Video Factory Areas for Improvement

  • Break up monolithic code base Asynchronous messaging based architecture Create supporting tools early Think about monitoring early

    Adapt old tools or implement new ones Continuous Delivery Smaller releases more often

    35 times to Live per week

    Video Factory A Quick Summary

  • AWS Enterprise Summit

    @AWS_UKI

    #AWSSummit